An infinite scroll can be a beautiful and functional way to present feed data. You can see ours on the homepage of artsy.net. It works by fetching a few items from the API, then fetching some more items as the user scrolls down the feed. Each API call returns the items along with a "cursor", which marks the position of the last item retrieved. Subsequent API calls include the cursor in the query string and the iteration resumes from there.
Why use a cursor and not standard pagination? Because inserting an item on top of the feed would shift the existing items down, causing the API to return a duplicate item on the page boundary. Removing an item from the top of the feed would pull the remaining items up, causing an item to be missed in the next request on the page boundary.
Today we're open-sourcing a small gem called mongoid-scroll, which implements this cursor-like behavior for MongoDB using mongoid or moped. Here's how it works.
Define a sample
FeedItem model with an index on
position. We'll be iterating over our feed, starting with the newest item first.
1 2 3 4 5 6 7 8
Insert some sample unordered data manufactured with faker.
1 2 3 4 5
Iterate over this collection using a cursor, 7 items at a time.
1 2 3 4 5 6 7 8 9 10 11 12
The result is, as expected, all 20 items in reverse order.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
We've used 4 queries to iterate over this collection.
The first ordered query without an existing cursor uses a
The last item returned has a position of 14 (we scrolled from 20 down to 14, including the boundaries).
Second and Third Query
The second ordered query has to fetch any item that comes after 14, including any other item that has the same position further in the same direction as the MongoDB order (there're no duplicates in our example, but it's entirely possible).
1 2 3 4
Note that we're sorting by
_id as well because MongoDB may relocate a document and therefore alter the natural order. See this commit for a test that reproduces this behavior.
We've chosen to break out of the loop after getting no data back in the 4th iteration. You can check whether the item retrieved is the last one in the collection as an alternative to prevent this fourth empty database query.
Cursors consist of the item's position and the item's BSON id. The cursor for the item at position 14 is
14:511d7c7c3b5552c92400000e. This cursor is parsed to construct the query on subsequent requests or can be supplied as a