Lucene data store and services

Lucene data store  will contain lines for each record and record file in the system. 

 All data in  Lucene data store  will be sent to Elastic search. Every time a record is updated an entry is made in  Lucene data store , and by default the data is sent synchroniously to ElasticSearch (fulltextBatchProcessBlobs). 

 Files are instead put in the  Lucene file queue . By default the indexer is notified imidiately, a will start executing the  Lucene file queue . When finished each translated text is written back into the  Lucene data store  and deleted from the  Lucene file queue . Data is by default be sent to Elastic search right away (fulltextBatchProcessFiles). 

 TS contains 2 services 

 

 Data index builder 

 File index builder 

 

 These services sends data from  Lucene data store  to ElasticSearch. As mentioned this will normally be carried out automatically / synchroniously, unless some kind of error occurs - like Elastic being offline etc. In that case unprocessed items queue up: In the datastore, file queue or both. 

 Running the services will handle everything in the queues. 

 Consider having  Data index builder running every day (1440 mins) to clean up the queue now and then.