nltk.corpus.reader.util.StreamBackedCorpusView

A 'view' of a corpus file, which acts like a sequence of tokens: it can be accessed by index, iterated over, etc. However, the tokens are only constructed as-needed -- the entire corpus is never stored in memory at once.

The constructor to StreamBackedCorpusView takes two arguments: a corpus fileid (specified as a string or as a PathPointer); and a block reader. A "block reader" is a function that reads zero or more tokens from a stream, and returns them as a list. A very simple example of a block reader is:

>>> def simple_block_reader(stream):
...     return stream.readline().split()

This simple block reader reads a single line at a time, and returns a single token (consisting of a string) for each whitespace-separated substring on the line.

When deciding how to define the block reader for a given corpus, careful consideration should be given to the size of blocks handled by the block reader. Smaller block sizes will increase the memory requirements of the corpus view's internal data structures (by 2 integers per block). On the other hand, larger block sizes may decrease performance for random access to the corpus. (But note that larger block sizes will not decrease performance for iteration.)

Internally, CorpusView maintains a partial mapping from token index to file position, with one entry per block. When a token with a given index i is requested, the CorpusView constructs it as follows:

1. First, it searches the toknum/filepos mapping for the token index closest to (but less than or equal to) i.
2. Then, starting at the file position corresponding to that index, it reads one block at a time using the block reader until it reaches the requested token.

The toknum/filepos mapping is created lazily: it is initially empty, but every time a new block is read, the block's initial token is added to the mapping. (Thus, the toknum/filepos map has one entry per block.)

In order to increase efficiency for random access patterns that have high degrees of locality, the corpus view may cache one or more blocks.