[ENG-10028] SHARE is not consistently indexing OSF content#11631
Open
mkovalua wants to merge 9 commits intoCenterForOpenScience:feature/pbs-26-2from
Open
[ENG-10028] SHARE is not consistently indexing OSF content#11631mkovalua wants to merge 9 commits intoCenterForOpenScience:feature/pbs-26-2from
mkovalua wants to merge 9 commits intoCenterForOpenScience:feature/pbs-26-2from
Conversation
mkovalua
force-pushed
the
fix/ENG-10028
branch
2 times, most recently
from
7486140 to
0d9d38b
Compare
mkovalua
changed the title
…h has_been_indexed None or False and public referents ...
mkovalua
force-pushed
the
fix/ENG-10028
branch
2 times, most recently
from
f0c33d4 to
54c5a9a
Compare
Comment on lines
+2577
to
+2578
| if isinstance(self, OsfStorageFile): | ||
| self.save(update_fields=['has_been_indexed'], skip_search=True) |
Collaborator
There was a problem hiding this comment.
What's the skip_search for?
Contributor
Author
There was a problem hiding this comment.
There is a recursion (another share update task is called) on save for files , such a way to avoid it
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ticket
Purpose
Content on the OSF is not consistently being SHARE indexed. Newly created content on the OSF does not seem to be consistently being automatically indexed in SHARE, some content seems to be indexed but other content is not. There does not seem to be any way to discern what content has been indexed vs not indexed . The inability of some content to be re-indexed in admin, this is causing significant issues for OSF users, their content is not Discoverable on the OSF.
Changes
Implementing admin page
/share_reindexto try to share reindex in background task public resources that is either failed(has_been_indexed=False)on last reindex attempt or have(has_been_indexed=None)https://openscience.atlassian.net/browse/ENG-10028?focusedCommentId=109746
223f-fa81-47eb-8968-b5ec8afb5bad.1.1.mp4
Side Effects
(
maybe it is possible to improve
get_not_indexed_guids_for_resource_with_no_indexed_guidin terms of queries1460d1b#diff-d288f5a6729b0d85cc2d5c82086b82324edbb94b369fac5f719e9fb01dbfc726L150-R171
mostly was related to GUID table store approach, maybe not relevant for current Mixins approach where no such a complex query
)
Shared thoughts about better solution (what better to use GUID table store approach / Mixins / separate table)
https://centerforopenscience.slack.com/archives/C03TPBCUC0K/p1773347865461569
for now I suppose Mixin is better option in terms of performance and usability
https://centerforopenscience.slack.com/archives/C03TPBCUC0K/p1773419614530719?thread_ts=1773347865.461569&cid=C03TPBCUC0K
For some reason not see Files to be reindexed into Share locally, maybe something is wrong with my local env,
did not change existing business logic
QE Notes
CE Notes
Documentation