[bioontology-support] Can't generate ChEBI mappings in my OntoPortal 3 VM

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[bioontology-support] Can't generate ChEBI mappings in my OntoPortal 3 VM

Miller, Mark
Currently, one of my main uses for BioPortal/OntoPortal is retrieving all mappings between ChEBI, DrOn, and RxNorm. I prefer to do that from the VM because it’s faster and I can load new versions as soon as they become available. And it’s better citizenship, right?

I recently upgraded to OntoPortal 3, and I can’t seem to get mappings to or from ChEBI.

Are there caches that can be cleared besides the four on the Admin page?

Is there some way to force a rebuild of the mappings? I have tried re-processing and re-annotating from the Ontology Administration page.

I have included a lot of attachments. I hope they are helpful and not too distracting. I can send anything else you might like to see.

Thanks,
Mark

—   —   —   



Here’s the Ontology Admin page:



I can retrieve mappings between RxNorm and STY, but I can’t get ChEBI mappings, at least through some of the web interfaces.



Although some mappings are available via the API… but they look stale, from ontologies I have deleted from the VM. I HAVE cleared all of the caches accessible from the Site Administration page.


[ ]

I have tried submitting ChEBI as both the OBO and OWL files, and as the full or lite edition.

I don’t see the words “error” or “warn” anywhere in the ChEBI parsing log. Here’s an excerpt:

Log file: CHEBI/1/parsing.log
# Logfile created on 2020-09-07 15:59:17 +0000 by logger.rb/v1.4.2
I, [2020-09-07T15:59:17.864488 #2539]  INFO -- : ["Starting to process http://data.bioontology.org/ontologies/CHEBI/submissions/1"]
I, [2020-09-07T15:59:17.868818 #2539]  INFO -- : ["Starting to process CHEBI/submissions/1"]
I, [2020-09-07T15:59:18.196865 #2539]  INFO -- : ["Java call [java -DentityExpansionLimit=2500000 -Xmx10240M -jar /srv/ontoportal/ncbo_cron/vendor/bundle/ruby/2.6.0/bundler/gems/ontologies_linked_data-613a5c836099/bin/owlapi-wrapper-1.3.6.jar -m /srv/ontoportal/data/repository/CHEBI/1/chebi_lite.obo -o /srv/ontoportal/data/repository/CHEBI/1 -r true]"]
I, [2020-09-07T16:00:12.628030 #2539]  INFO -- : ["2020-09-07T15:59:18 [main] INFO  o.s.n.o.OntologyParserCommand - Parsing invocation with values: ParserInvocation [inputRepositoryFolder=null, outputRepositoryFolder=/srv/ontoportal/data/repository/CHEBI/1, masterFileName=/srv/ontoportal/data/repository/CHEBI/1/chebi_lite.obo, invocationId=0, parserLog=, userReasoner= true]\n\n2020-09-07T15:59:18 [main] INFO  o.s.ncbo.oapiwrapper.OntologyParser - executor ...\n\n2020-09-07T15:59:19 [main] INFO  o.s.ncbo.oapiwrapper.OntologyParser - Input repository folder is null. Unique file being parsed.\n\n2020-09-07T15:59:19 [main] DEBUG o.e.rdf4j.rio.RDFParserRegistry - Registered service class org.eclipse.rdf4j.rio.binary.BinaryRDFParserFactory\n\n2020-09-07T15:59:19 [main] DEBUG o.e.rdf4j.rio.RDFParserRegistry - Registered service class org.eclipse.rdf4j.rio.n3.N3ParserFactory\n\n2020-09-07T15:59:19 [main] DEBUG o.e.rdf4j.rio.RDFParserRegistry - Registered service class org.eclipse.rdf4j.rio.nquads.NQuadsParserFactory\n\n2020-09-07T15:59:19 [main] DEBUG o.e.rdf4j.rio.RDFParserRegistry - Registered service class org.eclipse.rdf4j.rio.ntriples.NTriplesParserFactory\n\n2020-09-07T15:59:19 [main] DEBUG o.e.rdf4j.rio.RDFParserRegistry - Registered service class org.eclipse.rdf4j.rio.rdfjson.RDFJSONParserFactory\n\n2020-09-07T15:59:19 [main] DEBUG o.e.rdf4j.rio.RDFParserRegistry - Registered service class org.eclipse.rdf4j.rio.jsonld.JSONLDParserFactory\n\n2020-09-07T15:59:19 [main] DEBUG o.e.rdf4j.rio.RDFParserRegistry - Registered service class org.eclipse.rdf4j.rio.rdfxml.RDFXMLParserFactory\n\n2020-09-07T15:59:19 [main] DEBUG o.e.rdf4j.rio.RDFParserRegistry - Registered service class org.eclipse.rdf4j.rio.trix.TriXParserFactory\n\n2020-09-07T15:59:19 [main] DEBUG o.e.rdf4j.rio.RDFParserRegistry - Registered service class org.eclipse.rdf4j.rio.turtle.TurtleParserFactory\n\n2020-09-07T15:59:19 [main] DEBUG o.e.rdf4j.rio.RDFParserRegistry - Registered service class org.eclipse.rdf4j.rio.trig.TriGParserFactory\n\n2020-09-07T15:59:19 [main] DEBUG o.e.r.rio.DatatypeHandlerRegistry - Registered service class org.eclipse.rdf4j.rio.datatypes.XMLSchemaDatatypeHandler\n\n2020-09-07T15:59:19 [main] DEBUG o.e.r.rio.DatatypeHandlerRegistry - Registered service class org.eclipse.rdf4j.rio.datatypes.RDFDatatypeHandler\n\n2020-09-07T15:59:19 [main] DEBUG o.e.r.rio.DatatypeHandlerRegistry - Registered service class org.eclipse.rdf4j.rio.datatypes.DBPediaDatatypeHandler\n\n2020-09-07T15:59:19 [main] DEBUG o.e.r.rio.DatatypeHandlerRegistry - Registered service class org.eclipse.rdf4j.rio.datatypes.VirtuosoGeometryDatatypeHandler\n\n2020-09-07T15:59:19 [main] DEBUG o.e.r.rio.DatatypeHandlerRegistry - Registered service class org.eclipse.rdf4j.rio.datatypes.GeoSPARQLDatatypeHandler\n\n2020-09-07T15:59:19 [main] DEBUG o.e.r.rio.LanguageHandlerRegistry - Registered service class org.eclipse.rdf4j.rio.languages.RFC3066LanguageHandler\n\n2020-09-07T15:59:19 [main] DEBUG o.e.r.rio.LanguageHandlerRegistry - Registered service class org.eclipse.rdf4j.rio.languages.BCP47LanguageHandler\n\n2020-09-07T15:59:33 [main] INFO  o.s.ncbo.oapiwrapper.OntologyMetrics - Calculating metrics for /srv/ontoportal/data/repository/CHEBI/1/chebi_lite.obo\n\n2020-09-07T15:59:33 [main] INFO  o.s.ncbo.oapiwrapper.OntologyMetrics - Finished metrics calculation for /srv/ontoportal/data/repository/CHEBI/1/chebi_lite.obo in 57 milliseconds\n\n2020-09-07T15:59:33 [main] INFO  o.s.ncbo.oapiwrapper.OntologyMetrics - Generated metrics CSV file for /srv/ontoportal/data/repository/CHEBI/1/chebi_lite.obo\n\n2020-09-07T15:59:33 [main] INFO  o.s.ncbo.oapiwrapper.OntologyParser - Ontology document format: org.semanticweb.owlapi.formats.OBODocumentFormat\n\n2020-09-07T15:59:34 [main] INFO  o.s.ncbo.oapiwrapper.OntologyParser - isPrefixOWLOntologyFormat: false\n\n2020-09-07T15:59:38 [main] INFO  o.s.ncbo.oapiwrapper.OntologyParser - Adding version: 191\n\n2020-09-07T15:59:45 [main] INFO  o.s.ncbo.oapiwrapper.OntologyParser - isOBO: true\n\n2020-09-07T15:59:47 [main] INFO  o.s.ncbo.oapiwrapper.OntologyParser - Serializing ontology in RDF ...\n\n2020-09-07T16:00:12 [main] INFO  o.s.ncbo.oapiwrapper.OntologyParser - Serialization done!\n\n2020-09-07T16:00:12 [main] INFO  o.s.n.o.OntologyParserCommand - Parse result: true\n\n2020-09-07T16:00:12 [main] INFO  o.s.n.o.OntologyParserCommand - Output triples in: {}/srv/ontoportal/data/repository/CHEBI/1/owlapi.xrdf\n\n2020-09-07T16:00:12 [main] INFO  o.s.n.o.OntologyParserCommand - Finished parsing!\n"]
I, [2020-09-07T16:00:12.628684 #2539]  INFO -- : ["OWLAPI Java command: parsing finished successfully."]
I, [2020-09-07T16:00:12.630502 #2539]  INFO -- : ["Output size 193129660 in `/srv/ontoportal/data/repository/CHEBI/1/owlapi.xrdf`"]
I, [2020-09-07T16:01:16.993478 #2539]  INFO -- : ["Triples /srv/ontoportal/data/repository/CHEBI/1/owlapi.xrdf appended in <http://data.bioontology.org/ontologies/CHEBI/submissions/1>"]
I, [2020-09-07T16:01:18.487491 #2539]  INFO -- : ["1: page 1 of 58 - 2500 ontology terms retrieved in 1.22956868 sec."]
I, [2020-09-07T16:01:19.250110 #2539]  INFO -- : ["Asserting 2500 labels in <http://data.bioontology.org/ontologies/CHEBI/submissions/1>"]
I, [2020-09-07T16:01:19.451892 #2539]  INFO -- : ["Labels asserted in 0.199278285 sec."]
---snip---
I, [2020-09-07T16:03:13.058429 #2539]  INFO -- : ["1: page 58 of 58 - 2149 ontology terms retrieved in 0.919392601 sec."]
I, [2020-09-07T16:03:13.605387 #2539]  INFO -- : ["Asserting 2149 labels in <http://data.bioontology.org/ontologies/CHEBI/submissions/1>"]
I, [2020-09-07T16:03:13.810954 #2539]  INFO -- : ["Labels asserted in 0.204242249 sec."]
I, [2020-09-07T16:03:13.811189 #2539]  INFO -- : ["Asserting 4298 mappings in <http://data.bioontology.org/ontologies/CHEBI/submissions/1>"]
I, [2020-09-07T16:03:14.060541 #2539]  INFO -- : ["Mapping labels asserted in 0.246911618 sec."]
I, [2020-09-07T16:03:14.060640 #2539]  INFO -- : ["end generate_missing_labels traversed 144649 classes"]
I, [2020-09-07T16:03:14.060693 #2539]  INFO -- : ["Saved generated labels in /srv/ontoportal/data/repository/CHEBI/1/labels.ttl"]
I, [2020-09-07T16:03:14.060823 #2539]  INFO -- : ["Completed Missing Labels Generation: 1 in 116.91650822501106 sec. 144649 classes."]
I, [2020-09-07T16:03:14.677001 #2539]  INFO -- : ["Indexing ontology terms: CHEBI..."]
I, [2020-09-07T16:03:14.747152 #2539]  INFO -- : ["Removed ontology terms index (0.069995754s)"]
I, [2020-09-07T16:03:17.420513 #2539]  INFO -- : ["Thread 1: Page 1 of 145 - 1000 ontology terms retrieved in 1.0482e-05 sec."]
I, [2020-09-07T16:03:18.050454 #2539]  INFO -- : ["Thread 1: Page 1 of 145 attributes mapped in 0.629807943 sec."]
I, [2020-09-07T16:03:44.499101 #2539]  INFO -- : ["Thread 1: Page 1 of 145 - 1000 ontology terms indexed in 26.448257779 sec."]

---snip---
I, [2020-09-07T16:44:32.335034 #2539]  INFO -- : ["Thread 1: Page 145 of 145 - 649 ontology terms retrieved in 7.343e-06 sec."]
I, [2020-09-07T16:44:32.686103 #2539]  INFO -- : ["Thread 1: Page 145 of 145 attributes mapped in 0.350929584 sec."]
I, [2020-09-07T16:44:41.926707 #2539]  INFO -- : ["Thread 1: Page 145 of 145 - 649 ontology terms indexed in 9.240427009 sec."]
I, [2020-09-07T16:44:42.116333 #2539]  INFO -- : ["Ontology terms index commit in 0.188457243 sec."]
I, [2020-09-07T16:44:42.116492 #2539]  INFO -- : ["Completed indexing ontology terms: CHEBI in 2487.790310148004 sec. 144649 classes."]
I, [2020-09-07T16:44:42.238956 #2539]  INFO -- : ["Indexing ontology properties: CHEBI..."]
I, [2020-09-07T16:44:42.261297 #2539]  INFO -- : ["Removed ontology properties index in 0.02218551 seconds."]
I, [2020-09-07T16:44:42.462311 #2539]  INFO -- : ["Indexing a total of 1 pages of 2500 properties each."]
I, [2020-09-07T16:44:42.524558 #2539]  INFO -- : ["Page 1 of ontology properties indexed in 0.062042124 seconds."]
I, [2020-09-07T16:44:42.547796 #2539]  INFO -- : ["Ontology properties index commit in 0.02297986 seconds."]
I, [2020-09-07T16:44:42.547940 #2539]  INFO -- : ["Completed indexing ontology properties of CHEBI in 0.30903106299228966 sec. Total of 38 properties indexed."]
I, [2020-09-07T16:44:42.666989 #2539]  INFO -- : ["metrics_for_submission start"]
I, [2020-09-07T16:45:45.772482 #2539]  INFO -- : ["Metrics groupby_children retrieved 12321 in 4.306272875 sec."]
I, [2020-09-07T16:45:45.777303 #2539]  INFO -- : ["Metrics count_classes retrieved 144649 in 0.000317321 sec."]
I, [2020-09-07T16:45:46.325177 #2539]  INFO -- : ["Metrics count cls with def 118708 in 0.547730875 sec."]
I, [2020-09-07T16:45:46.325947 #2539]  INFO -- : ["Class metrics finished in 63.658839633 sec."]
I, [2020-09-07T16:45:46.325989 #2539]  INFO -- : ["class_metrics finished"]
I, [2020-09-07T16:45:46.326441 #2539]  INFO -- : ["individuals finished"]
I, [2020-09-07T16:45:46.326651 #2539]  INFO -- : ["properties finished"]
I, [2020-09-07T16:45:46.326961 #2539]  INFO -- : ["generation of metrics file finished"]
I, [2020-09-07T16:45:46.564402 #2539]  INFO -- : ["Bubastis diff: no older submissions available for http://data.bioontology.org/ontologies/CHEBI/submissions/1."]
I, [2020-09-07T16:45:46.569379 #2539]  INFO -- : ["Submission processing of http://data.bioontology.org/ontologies/CHEBI/submissions/1 completed successfully"]
D, [2020-09-07T16:45:46.605989 #2539] DEBUG -- : ["Archiving submissions previous to http://data.bioontology.org/ontologies/CHEBI/submissions/1..."]
D, [2020-09-07T16:45:46.623654 #2539] DEBUG -- : ["Completed archiving submissions previous to http://data.bioontology.org/ontologies/CHEBI/submissions/1"]
I, [2020-09-07T16:45:46.646961 #2539]  INFO -- : [["Caching classes of CHEBI"]]
I, [2020-09-07T16:45:46.647023 #2539]  INFO -- : ["Caching classes of CHEBI"]
I, [2020-09-07T16:45:48.748482 #2539]  INFO -- : [["Page 1 of 58 - 2500 classes retrieved in 2.101163434 sec."]]
I, [2020-09-07T16:45:48.748509 #2539]  INFO -- : ["Page 1 of 58 - 2500 classes retrieved in 2.101163434 sec."]
I, [2020-09-07T16:45:50.456922 #2539]  INFO -- : [["Page 1 of 58 cached in Annotator in 1.708290642 sec."]]
I, [2020-09-07T16:45:50.456954 #2539]  INFO -- : ["Page 1 of 58 cached in Annotator in 1.708290642 sec."]
---snip---
I, [2020-09-07T16:49:11.988457 #2539]  INFO -- : ["Page 57 of 58 cached in Annotator in 1.358687309 sec."]
I, [2020-09-07T16:49:14.137903 #2539]  INFO -- : [["Page 58 of 58 - 2149 classes retrieved in 2.149309102 sec."]]
I, [2020-09-07T16:49:14.137929 #2539]  INFO -- : ["Page 58 of 58 - 2149 classes retrieved in 2.149309102 sec."]
I, [2020-09-07T16:49:15.348864 #2539]  INFO -- : [["Page 58 of 58 cached in Annotator in 1.210797795 sec."]]
I, [2020-09-07T16:49:15.348895 #2539]  INFO -- : ["Page 58 of 58 cached in Annotator in 1.210797795 sec."]
I, [2020-09-07T16:49:15.443946 #2539]  INFO -- : [["Completed caching ontology: CHEBI (http://data.bioontology.org/ontologies/CHEBI/submissions/1) in 208.70213225399493 sec. 144649 classes."]]
I, [2020-09-07T16:49:15.444113 #2539]  INFO -- : ["Completed caching ontology: CHEBI (http://data.bioontology.org/ontologies/CHEBI/submissions/1) in 208.70213225399493 sec. 144649 classes."]
D, [2020-09-07T16:50:16.528838 #2539] DEBUG -- : ["Completed processing of http://data.bioontology.org/ontologies/CHEBI/submissions/1 in 3058.8s"]
I, [2020-09-07T16:50:16.544463 #2539]  INFO -- : ["Running ontologies report for ontologies CHEBI...\n"]
I, [2020-09-07T16:50:16.544525 #2539]  INFO -- : ["Processing report for CHEBI - 1 of 1 ontologies."]
I, [2020-09-07T16:50:17.648376 #2539]  INFO -- : ["Finished report for CHEBI in 1.103617781001958 sec."]
I, [2020-09-07T16:50:17.651608 #2539]  INFO -- : ["Finished updating report for ontologies CHEBI. Wrote report data to /srv/ontoportal/data/reports/ontologies_report.json.\n"]
I, [2020-09-07T18:54:17.684928 #8073]  INFO -- : ["Starting to process http://data.bioontology.org/ontologies/CHEBI/submissions/1"]
I, [2020-09-07T18:54:17.687380 #8073]  INFO -- : ["Starting to process CHEBI/submissions/1"]
I, [2020-09-07T18:54:17.727812 #8073]  INFO -- : ["Submission processing of http://data.bioontology.org/ontologies/CHEBI/submissions/1 completed successfully"]
I, [2020-09-07T18:54:17.821911 #8073]  INFO -- : [["Caching classes of CHEBI"]]
I, [2020-09-07T18:54:17.821944 #8073]  INFO -- : ["Caching classes of CHEBI"]
I, [2020-09-07T18:54:18.688614 #8073]  INFO -- : [["Page 1 of 58 - 2500 classes retrieved in 0.865643534 sec."]]
I, [2020-09-07T18:54:18.688640 #8073]  INFO -- : ["Page 1 of 58 - 2500 classes retrieved in 0.865643534 sec."]
I, [2020-09-07T18:54:20.256994 #8073]  INFO -- : [["Page 1 of 58 cached in Annotator in 1.568235775 sec."]]
I, [2020-09-07T18:54:20.257039 #8073]  INFO -- : ["Page 1 of 58 cached in Annotator in 1.568235775 sec."]
---snip---
I, [2020-09-07T18:56:23.914798 #8073]  INFO -- : [["Page 58 of 58 - 2149 classes retrieved in 0.669476044 sec."]]
I, [2020-09-07T18:56:23.914825 #8073]  INFO -- : ["Page 58 of 58 - 2149 classes retrieved in 0.669476044 sec."]
I, [2020-09-07T18:56:25.117156 #8073]  INFO -- : [["Page 58 of 58 cached in Annotator in 1.20218612 sec."]]
I, [2020-09-07T18:56:25.117186 #8073]  INFO -- : ["Page 58 of 58 cached in Annotator in 1.20218612 sec."]
I, [2020-09-07T18:56:25.224264 #8073]  INFO -- : [["Completed caching ontology: CHEBI (http://data.bioontology.org/ontologies/CHEBI/submissions/1) in 127.29543524500332 sec. 144649 classes."]]
I, [2020-09-07T18:56:25.224325 #8073]  INFO -- : ["Completed caching ontology: CHEBI (http://data.bioontology.org/ontologies/CHEBI/submissions/1) in 127.29543524500332 sec. 144649 classes."]
D, [2020-09-07T18:57:25.981082 #8073] DEBUG -- : ["Completed processing of http://data.bioontology.org/ontologies/CHEBI/submissions/1 in 188.47s"]
I, [2020-09-07T18:57:25.982870 #8073]  INFO -- : ["Running ontologies report for ontologies CHEBI...\n"]
I, [2020-09-07T18:57:25.982940 #8073]  INFO -- : ["Processing report for CHEBI - 1 of 1 ontologies."]
I, [2020-09-07T18:57:26.504996 #8073]  INFO -- : ["Finished report for CHEBI in 0.5218716039962601 sec."]
I, [2020-09-07T18:57:26.507349 #8073]  INFO -- : ["Finished updating report for ontologies CHEBI. Wrote report data to /srv/ontoportal/data/reports/ontologies_report.json.\n"]
OntoPortal Appliance 3.0.2  Powered by BioPortal  Projects

_______________________________________________
bioontology-support mailing list
[hidden email]
https://mailman.stanford.edu/mailman/listinfo/bioontology-support
Reply | Threaded
Open this post in threaded view
|

Re: [bioontology-support] Can't generate ChEBI mappings in my OntoPortal 3 VM

jvendetti
Hello Mark,


On Sep 9, 2020, at 6:45 AM, Miller, Mark <[hidden email]> wrote:

Currently, one of my main uses for BioPortal/OntoPortal is retrieving all mappings between ChEBI, DrOn, and RxNorm. I prefer to do that from the VM because it’s faster and I can load new versions as soon as they become available. And it’s better citizenship, right?


Yes, thank you!



I recently upgraded to OntoPortal 3, and I can’t seem to get mappings to or from ChEBI.


I have OntoPortal version 3.0.2 installed on my laptop. I uploaded CHEBI and RXNORM, and was able to reproduce this issue. 

I can retrieve mappings for individual classes, but not ontology level mappings. In other words, this call to get all mappings between RXNORM and CHEBI returns an empty set:


… but calls to get mappings for an individual class like “cystine” do return results:




Are there caches that can be cleared besides the four on the Admin page?


No, that covers clearing of all the caches.



Is there some way to force a rebuild of the mappings? I have tried re-processing and re-annotating from the Ontology Administration page.


I had to reprocess both ontologies because initially I hadn’t allocated enough memory to the virtual machine. It doesn’t appear that reprocessing addresses this issue.

Apologies for the inconvenience, but I’ll have to continue to troubleshoot on our end to see if I can find the root cause of the issue. I’ll update you when I know something more.

Kind regards,
Jennifer




_______________________________________________
bioontology-support mailing list
[hidden email]
https://mailman.stanford.edu/mailman/listinfo/bioontology-support
Reply | Threaded
Open this post in threaded view
|

Re: [bioontology-support] [External] Can't generate ChEBI mappings in my OntoPortal 3 VM

Miller, Mark
Thanks for the quick reply.

How much memory do you recommend for these parse jobs? I’ll be doing DrOn soon, too. How could I tell if I’m not allocating enough memory?

Can you point me towards documentation about triggering reprocessing? I think I saw it before, but now that the documentation is so much more comprehensive, it can be hard to find things! 

I have loaded the Consumer Health Vocabulary Ontology (OCHV) since I sent the first message and am characterizing the mapping success now.

Thanks,
Mark

On Sep 9, 2020, at 5:06 PM, Jennifer Leigh Vendetti <[hidden email]> wrote:

Hello Mark,


On Sep 9, 2020, at 6:45 AM, Miller, Mark <[hidden email]> wrote:

Currently, one of my main uses for BioPortal/OntoPortal is retrieving all mappings between ChEBI, DrOn, and RxNorm. I prefer to do that from the VM because it’s faster and I can load new versions as soon as they become available. And it’s better citizenship, right?


Yes, thank you!



I recently upgraded to OntoPortal 3, and I can’t seem to get mappings to or from ChEBI.


I have OntoPortal version 3.0.2 installed on my laptop. I uploaded CHEBI and RXNORM, and was able to reproduce this issue. 

I can retrieve mappings for individual classes, but not ontology level mappings. In other words, this call to get all mappings between RXNORM and CHEBI returns an empty set:


… but calls to get mappings for an individual class like “cystine” do return results:




Are there caches that can be cleared besides the four on the Admin page?


No, that covers clearing of all the caches.



Is there some way to force a rebuild of the mappings? I have tried re-processing and re-annotating from the Ontology Administration page.


I had to reprocess both ontologies because initially I hadn’t allocated enough memory to the virtual machine. It doesn’t appear that reprocessing addresses this issue.

Apologies for the inconvenience, but I’ll have to continue to troubleshoot on our end to see if I can find the root cause of the issue. I’ll update you when I know something more.

Kind regards,
Jennifer





_______________________________________________
bioontology-support mailing list
[hidden email]
https://mailman.stanford.edu/mailman/listinfo/bioontology-support
Reply | Threaded
Open this post in threaded view
|

Re: [bioontology-support] [External] Can't generate ChEBI mappings in my OntoPortal 3 VM

jvendetti
Hi Mark,


On Sep 9, 2020, at 2:14 PM, Miller, Mark <[hidden email]> wrote:

How much memory do you recommend for these parse jobs?


There are some general system requirements documented here:

https://ontoportal.github.io/administration/general/system_requirements/

However, it depends on the size of the ontology. We recently corresponded with someone that uploaded SNOMED CT (roughly 1.47 GB ontology source file), and they needed to allocate 32 GB to the appliance. Anything less resulted in out of memory errors.

For my installation, I have 28 GB allocated, which is the maximum my development laptop can handle without keeling over. For smaller ontologies, you wouldn’t need as much.


I’ll be doing DrOn soon, too. How could I tell if I’m not allocating enough memory?


If processing fails due to an out of memory error, there could be a message to that effect in the parsing log file for the ontology. The Admin -> Ontology Administration page has a URL column with “Log” links that allow you to view the parsing log file. Just make sure to look at the timestamps for the processing, as multiple runs of processing the same ontology will output log data into that same file.

Alternatively, if you have access to the administration console for the appliance, you can navigate to the submission directory on the command line and view the parsing log in a text editor:




Can you point me towards documentation about triggering reprocessing? I think I saw it before, but now that the documentation is so much more comprehensive, it can be hard to find things! 


Sure, no problem. 

From the OntoPortal application, you can kick off a reprocessing of an ontology using the Admin -> Ontology Administration page. You would select the row for the desired ontology, choose “Process" from the “Apply to Selected Rows” dropdown box, then click the Go button. 

Alternatively, you can use these instructions to reprocess the ontology from the command line:


I use this approach because I can watch the log output and see immediately if errors are written out.


I have loaded the Consumer Health Vocabulary Ontology (OCHV) since I sent the first message and am characterizing the mapping success now.

It’s not apparent to me that this problem is specific to any one ontology. I’ve uploaded five separate ontologies, and there are no ontology-level mappings for any of them.

Kind regards,
Jennifer


_______________________________________________
bioontology-support mailing list
[hidden email]
https://mailman.stanford.edu/mailman/listinfo/bioontology-support
Reply | Threaded
Open this post in threaded view
|

Re: [bioontology-support] Can't generate ChEBI mappings in my OntoPortal 3 VM

jvendetti
In reply to this post by Miller, Mark
Hi Mark,


On Sep 9, 2020, at 6:45 AM, Miller, Mark <[hidden email]> wrote:

Currently, one of my main uses for BioPortal/OntoPortal is retrieving all mappings between ChEBI, DrOn, and RxNorm. I prefer to do that from the VM because it’s faster and I can load new versions as soon as they become available. And it’s better citizenship, right?

I recently upgraded to OntoPortal 3, and I can’t seem to get mappings to or from ChEBI.


OK, I’ve tracked down the underlying reason that you’re seeing this behavior.

In order to retrieve all of the mappings between two ontologies, the system depends on the existence of a persistent count of the number of mappings for each ontology. You can see these persisted counts using a REST call like the following:


These counts are generated and stored as part of a cron job that is configured to run once every Saturday, i.e.:


The historical reasons for this have to do with the 4store backend triplestore that is used in OntoPortal’s stack. Unfortunately, COUNT queries in 4store aren’t performant and we’ve had to add some workarounds in the system that prevent these types of queries from happening frequently in the production BioPortal that we host at Stanford. These cron job settings were carried over into our OntoPortal virtual appliance distribution.

All of that is to say that you have 2 options for seeing the mappings between ontologies in your appliance:

1). The easiest option is to wait until after the cron job has executed on Sat. at 12:30am, after which time you’ll be able to see all of the mappings between ontologies. (You may have to clear the caches via the Admin page).

2). If you have an urgent need to see the mappings, you can go the more difficult route and manually modify the schedule for this cron job. To follow are the instructions for how you would do this from the administration console:

a). Change to the ontoportal user:

sudo su - ontoportal

b). Shut down the current running cron process:

cd /srv/ontoportal/ncbo_cron/
bin/ncbo_cron —kill

c). Edit the crontab schedule for the “create_mapping_counts” job on line 56 of the configuration file (/srv/ontoportal/ncbo_cron/lib/ncbo_cron/config.rb). This page is helpful for showing examples of crontab schedule syntax: https://crontab.guru/examples.html.

d). After editing / saving / closing the configuration file, reboot your appliance.

After rebooting your appliance, the cron job scheduler will have picked up your new setting on how often you’d like to run the cron_mapping_counts job. If you want to check the status of whether the job is running successfully, the log output is stored here: /srv/ontoportal/ncbo_cron/logs/scheduler-mapping-counts.log.

Kind regards,
Jennifer



 



_______________________________________________
bioontology-support mailing list
[hidden email]
https://mailman.stanford.edu/mailman/listinfo/bioontology-support
Reply | Threaded
Open this post in threaded view
|

Re: [bioontology-support] [External] Can't generate ChEBI mappings in my OntoPortal 3 VM

Miller, Mark
In reply to this post by jvendetti
The mappings in my OntoPortal VM have indeed resolved themselves.  Thanks.

-Mark


On Sep 9, 2020, at 7:44 PM, Jennifer Leigh Vendetti <[hidden email]> wrote:

Hi Mark,


On Sep 9, 2020, at 2:14 PM, Miller, Mark <[hidden email]> wrote:

How much memory do you recommend for these parse jobs?


There are some general system requirements documented here:

https://ontoportal.github.io/administration/general/system_requirements/

However, it depends on the size of the ontology. We recently corresponded with someone that uploaded SNOMED CT (roughly 1.47 GB ontology source file), and they needed to allocate 32 GB to the appliance. Anything less resulted in out of memory errors.

For my installation, I have 28 GB allocated, which is the maximum my development laptop can handle without keeling over. For smaller ontologies, you wouldn’t need as much.


I’ll be doing DrOn soon, too. How could I tell if I’m not allocating enough memory?


If processing fails due to an out of memory error, there could be a message to that effect in the parsing log file for the ontology. The Admin -> Ontology Administration page has a URL column with “Log” links that allow you to view the parsing log file. Just make sure to look at the timestamps for the processing, as multiple runs of processing the same ontology will output log data into that same file.

Alternatively, if you have access to the administration console for the appliance, you can navigate to the submission directory on the command line and view the parsing log in a text editor:




Can you point me towards documentation about triggering reprocessing? I think I saw it before, but now that the documentation is so much more comprehensive, it can be hard to find things! 


Sure, no problem. 

From the OntoPortal application, you can kick off a reprocessing of an ontology using the Admin -> Ontology Administration page. You would select the row for the desired ontology, choose “Process" from the “Apply to Selected Rows” dropdown box, then click the Go button. 

Alternatively, you can use these instructions to reprocess the ontology from the command line:


I use this approach because I can watch the log output and see immediately if errors are written out.


I have loaded the Consumer Health Vocabulary Ontology (OCHV) since I sent the first message and am characterizing the mapping success now.

It’s not apparent to me that this problem is specific to any one ontology. I’ve uploaded five separate ontologies, and there are no ontology-level mappings for any of them.

Kind regards,
Jennifer



_______________________________________________
bioontology-support mailing list
[hidden email]
https://mailman.stanford.edu/mailman/listinfo/bioontology-support