Repository navigation
older style structured properties issue #129
Description
There is an additional issue with py27 ndb models with dotted property names.
Related to #126
In the case of a structured property which has repeated properties of different lengths, only the first two items are loaded.
For example:
{
"int_prop": 42,
"struct_prop.items.one": ["first", "first-1"],
"struct_prop.items.two": ["second", "second-1"],
"struct_prop.items.three": ["third", "third-1", "third-2"]
}
Current result (py37 python-ndb):
TestRepeatedStructuredModel(key=Key('TestRepeatedStructuredModel', '123456'), int_prop=42, struct_prop=StructuredRepeatedPropModel(key=Key('StructuredRepeatedPropModel', None), items=[StructuredPropModel(key=Key('StructuredPropModel', None), one='first', three='third', two='second'), StructuredPropModel(key=Key('StructuredPropModel', None), one='first-1', three='third-1', two='second-1')]))
Expected result (py27 ndb):
TestRepeatedStructuredModel(key=Key('TestRepeatedStructuredModel', '123456'), int_prop=42, struct_prop=StructuredRepeatedPropModel(key=Key('StructuredRepeatedPropModel', None), items=[StructuredPropModel(key=Key('StructuredPropModel', None), one='first', three='third', two='second'), StructuredPropModel(key=Key('StructuredPropModel', None), one='first-1', three='third-1', two='second-1'), StructuredPropModel(key=Key('StructuredPropModel', None), one=None, three='third-2', two=None)]))
All working as expected, if all dotted repeated properties have the same length
{
"int_prop": 42,
"struct_prop.items.one": ["first", "first-1"],
"struct_prop.items.two": ["second", "second-1"],
"struct_prop.items.three": ["third", "third-1"]
}
Entity Protobuf:
{
partition_id {
project_id: "kuk"
}
path {
kind: "TestRepeatedStructuredModel"
name: "123456"
}
}
properties {
key: "int_prop"
value {
integer_value: 42
}
}
properties {
key: "struct_prop.items.one"
value {
array_value {
values {
string_value: "first"
}
values {
string_value: "first-1"
}
}
}
}
properties {
key: "struct_prop.items.three"
value {
array_value {
values {
string_value: "third"
}
values {
string_value: "third-1"
}
values {
string_value: "third-2"
}
}
}
}
properties {
key: "struct_prop.items.two"
value {
array_value {
values {
string_value: "second"
}
values {
string_value: "second-1"
}
}
}
}
Test example:
@ndb_client_context()
def test_entity_with_dotted_repeated_structured_property_different_length(self):
db_driver = DbDriver(kind="TestRepeatedStructuredModel")
ds_key = '123456'
data = {
"int_prop": 42,
"struct_prop.items.one": ["first", "first-1"],
"struct_prop.items.two": ["second", "second-1"],
"struct_prop.items.three": ["third", "third-1", "third-2"]
}
db_driver.put(ds_key, data)
key = ndb.Key(TestRepeatedStructuredModel, ds_key)
retrieved = key.get()
print(f'retrieved: {retrieved}')
assert retrieved.int_prop == 42
assert len(retrieved.struct_prop.items) == 3 // Error: Returning 2 items
assert key.delete() is None
assert key.get() is None
db_driver.clear()
Activity
chrisrossi commentedon Jul 2, 2019
Looking at this. Thanks.
chrisrossi commentedon Jul 2, 2019
Hi @alexrwix ,
I'm trying to understand how or why this would come up.
In an interactive console, I've used the py27 version of NDB to create and store an entity:
The entity it creates in Datastore is:
As you can see, missing values for subproperties are filled in with
nullwhich is what I would intuitively expect. If it were to actually store lists of uneven length, it would be indeterminate which property values are supposed to go in which subentity. For example, let's say it had actually elided thenullvalues and stored this:That would get interpreted as having a single subentity where
one,two, andthreeare"Hamlet","Julius Caesar", and"Othello"respectively, which is not what we were trying to store.Hewing more closely to your example:
We get:
Because the nulls are included, this should work just fine with the code as-is.
Consider this variant:
Which gets us:
Here, if we were to elide the nulls such that
foo.oneandfoo.twoeach only had two values, then we'd get back the wrong subentities when we tried to decode.Is there a real-world use case I'm missing where NDB would be writing the subentities without
nullspacers? If so, how does that happen, and what's the argument for assumingnullpaddings at the end of the shorter sequences when it seems like they could go anywhere in the sequence?alexrwix commentedon Jul 2, 2019
Hi Chris.
Yes of course.
I'm checking python-ndb with real production content.
Our repeated structured property models inherited from ndb.Expando so the same models sometimes have a different set of properties.
Below is an example of real datastore entity
output property is:
Here is an entitiy example loaded directly from datastore:
After loading with py37 python-ndb, output items array has 8 items instead of with 12
But the biggest problem is that this issue is INCONSISTENT!
It depends on properties order in the entity that returned from datastore!
In case of this properties order, all items loaded correctly
chrisrossi commentedon Jul 3, 2019
So, I've tweaked my py2.7 test script to use
Expandoand leavethreeundeclared, to be handled by theExpandoclass:Doing this, I do get an entity with uneven repeated property lengths, like you describe:
The part that I'm still having trouble with is the fact that the subentities, in effect, are getting scrambled. If we just naively pad the end of the shorter property with nulls, we'll get the following subentities:
This, obviously, isn't what we tried to store and as far as I can tell is a bug, not a feature to be replicated. Unless I am missing something, the sub-entities in your production database are going to be somewhat mixed up with properties that were saved as being part of one sub-entity, being assigned to a different sub-entity upon retrieval.
Do you see what the problem is here? Is there something I'm missing?
alexrwix commentedon Jul 3, 2019
Hi Chris.
Yes. This specific example looks very problematic.
It really looks like py27 ndb bug.
In the current py37 version, we don't have such problem because it's saved as an embedded property.
Am I right?
I have never tried such scenario.
In our case, all "SubKind" constant properties always have values.
What is your suggestion?
How loading problem can be resolved?
We have hundreds of millions of records with such properties.
chrisrossi commentedon Jul 3, 2019
Correct.
As far as I can tell, your example is such a scenario.
In your example,
qualityhas 8 values buturlhas 12 values. Therefore, there are 4 items that don't havequalityvalues. The issue is that we have no way of knowing which 4 items. We can no longer trust thatqualityvalues match up to the correct items upon marshalling from Datastore. We don't have enough data to figure that out. This is what I mean by your item data being "scrambled".That is a good question. I don't know yet. We probably need to have some internal discussion. I believe I understand the problem, but want to discuss with my team what the best approach for a solution might be.
alexrwix commentedon Jul 4, 2019
Thank you.
Looking forward to your solution.
10 remaining items
Read legacy data with Repeated Structured Expando properties.
Read legacy data with Repeated Structured Expando properties.
Read legacy data with Repeated Structured Expando properties. (#176)
fix(model): Ensure repeated props have same kind when converting from ds
fix(model): Ensure repeated props have same kind when converting from ds
fix(model): Ensure repeated props have same kind when converting from…