Import failures

[pombredanne]

Using this branch https://github.com/nexB/vulnerablecode/tree/misc-updates (with exception catching and enhanced logging) I get these errors:

$ DJANGO_DEV=1 python manage.py import --all  --verbosity 2
Importing data from rust
Successfully imported data from rust
Importing data from alpine
Successfully imported data from alpine
Importing data from archlinux
Successfully imported data from archlinux
Importing data from debian
Successfully imported data from debian
Importing data from npm
Failed to _get_or_create_vulnerability: {'summary': 'attackers can write arbitrary files when a malicious archive is extracted.'}:
Traceback (most recent call last):
  File "/tmp/vulnerablecode/vulnerabilities/import_runner.py", line 256, in _get_or_create_vulnerability
    vuln, created = models.Vulnerability.objects.get_or_create(**query_kwargs)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/models/query.py", line 559, in get_or_create
    return self.get(**kwargs), False
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/models/query.py", line 419, in get
    raise self.model.MultipleObjectsReturned(
vulnerabilities.models.Vulnerability.MultipleObjectsReturned: get() returned more than one Vulnerability -- it returned 2!

Failed to process advisory: Advisory(summary='attackers can write arbitrary files when a malicious archive is extracted.', impacted_package_urls={PackageURL(type='npm', namespace=None, name='decompress-zip', version='0.3.0', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='decompress-zip', version='0.2.1', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='decompress-zip', version='0.2.0', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='decompress-zip', version='0.0.4', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='decompress-zip', version='0.0.7', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='decompress-zip', version='0.0.3', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='decompress-zip', version='0.0.8', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='decompress-zip', version='0.0.5', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='decompress-zip', version='0.1.0', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='decompress-zip', version='0.0.6', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='decompress-zip', version='0.0.2', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='decompress-zip', version='0.3.1', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='decompress-zip', version='0.0.1', qualifiers=OrderedDict(), subpath=None)}, resolved_package_urls={PackageURL(type='npm', namespace=None, name='decompress-zip', version='0.2.2', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='decompress-zip', version='0.3.2', qualifiers=OrderedDict(), subpath=None)}, vuln_references=[Reference(url='https://registry.npmjs.org/-/npm/v1/advisories/488', reference_id=488)], cve_id=''):
Traceback (most recent call last):
  File "/tmp/vulnerablecode/vulnerabilities/import_runner.py", line 146, in process_advisories
    vuln, vuln_created = _get_or_create_vulnerability(advisory)
  File "/tmp/vulnerablecode/vulnerabilities/import_runner.py", line 256, in _get_or_create_vulnerability
    vuln, created = models.Vulnerability.objects.get_or_create(**query_kwargs)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/models/query.py", line 559, in get_or_create
    return self.get(**kwargs), False
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/models/query.py", line 419, in get
    raise self.model.MultipleObjectsReturned(
vulnerabilities.models.Vulnerability.MultipleObjectsReturned: get() returned more than one Vulnerability -- it returned 2!

Failed to _get_or_create_vulnerability: {'summary': 'attackers can write arbitrary files when a malicious archive is extracted.'}:
Traceback (most recent call last):
  File "/tmp/vulnerablecode/vulnerabilities/import_runner.py", line 256, in _get_or_create_vulnerability
    vuln, created = models.Vulnerability.objects.get_or_create(**query_kwargs)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/models/query.py", line 559, in get_or_create
    return self.get(**kwargs), False
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/models/query.py", line 419, in get
    raise self.model.MultipleObjectsReturned(
vulnerabilities.models.Vulnerability.MultipleObjectsReturned: get() returned more than one Vulnerability -- it returned 2!

Failed to process advisory: Advisory(summary='attackers can write arbitrary files when a malicious archive is extracted.', impacted_package_urls={PackageURL(type='npm', namespace=None, name='bower', version='1.7.6', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.4.0', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.3.1', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.2.8', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.8.6', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.3.11', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.3.2', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.6.6', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.6.4', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.3.1', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.2.5', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.9.0', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.3.2', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.5.0', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.4.0', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.3.7', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.7.9', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.1.0', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.8.0', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.1.1', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.1.2', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.5.1', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.2.6', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.7.1', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.6.5', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.7.8', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.6.1', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.5.0', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.2.0', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.8.2', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.4.1', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.2.3', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.2.0', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.3.0', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.5.1', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.6.6', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.6.3', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.8.3', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.6.2', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.6.0', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.0.1', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.2.7', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.3.10', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.5.3', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.6.5', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.7.0', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.8.5', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.1.0', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.7.1', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.7.5', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.2.4', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.3.0', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.8.3', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.5.2', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.0.3', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.3.4', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.6.4', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.1.2', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.8.4', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.1.3', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.3.9', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.6.7', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.2.1', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.0.0', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.2.2', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.3.5', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.7.10', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.3.12', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.6.9', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.7.0', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.8.4', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.10.0', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.3.8', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.7.2', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.3.6', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.9.2', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.6.8', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.9.1', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.0.2', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.3.3', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.6.2', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.6.7', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.5.4', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.8.1', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.6.3', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.6.8', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.4.2', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.8.2', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.8.6', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='0.8.0', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.7.7', qualifiers=OrderedDict(), subpath=None)}, resolved_package_urls={PackageURL(type='npm', namespace=None, name='bower', version='1.8.8', qualifiers=OrderedDict(), subpath=None), PackageURL(type='npm', namespace=None, name='bower', version='1.8.7', qualifiers=OrderedDict(), subpath=None)}, vuln_references=[Reference(url='https://registry.npmjs.org/-/npm/v1/advisories/487', reference_id=487)], cve_id=''):
Traceback (most recent call last):
  File "/tmp/vulnerablecode/vulnerabilities/import_runner.py", line 146, in process_advisories
    vuln, vuln_created = _get_or_create_vulnerability(advisory)
  File "/tmp/vulnerablecode/vulnerabilities/import_runner.py", line 256, in _get_or_create_vulnerability
    vuln, created = models.Vulnerability.objects.get_or_create(**query_kwargs)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/models/query.py", line 559, in get_or_create
    return self.get(**kwargs), False
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/models/query.py", line 419, in get
    raise self.model.MultipleObjectsReturned(
vulnerabilities.models.Vulnerability.MultipleObjectsReturned: get() returned more than one Vulnerability -- it returned 2!

Successfully imported data from npm
Importing data from ruby
Successfully imported data from ruby
Importing data from ubuntu
Failed to get updated_advisories: <xml.etree.ElementTree.ElementTree object at 0x7f5fd9fec670> with {'type': 'deb', 'namespace': 'ubuntu'}:
Traceback (most recent call last):
  File "/tmp/vulnerablecode/vulnerabilities/data_source.py", line 453, in updated_advisories
    oval_data = self.get_data_from_xml_doc(oval_file, metadata)
  File "/tmp/vulnerablecode/vulnerabilities/data_source.py", line 482, in get_data_from_xml_doc
    raw_data = oval_doc.get_data()
  File "/tmp/vulnerablecode/vulnerabilities/oval_parser.py", line 76, in get_data
    test_data['version_ranges'] = self.get_versionsrngs_from_state(
  File "/tmp/vulnerablecode/vulnerabilities/oval_parser.py", line 152, in get_versionsrngs_from_state
    return RangeSpecifier(version_range)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/dephell_specifier/range_specifier.py", line 40, in __init__
    self._specs = self._parse(spec)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/dephell_specifier/range_specifier.py", line 62, in _parse
    result.add(cls._parse_star_and_operator(constr))
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/dephell_specifier/range_specifier.py", line 123, in _parse_star_and_operator
    parts = version.release[:-1] + (version.release[-1] + 1, )
TypeError: 'NoneType' object is not subscriptable

Failed to get updated_advisories: <xml.etree.ElementTree.ElementTree object at 0x7f5fe7f61d00> with {'type': 'deb', 'namespace': 'ubuntu'}:
Traceback (most recent call last):
  File "/tmp/vulnerablecode/vulnerabilities/data_source.py", line 453, in updated_advisories
    oval_data = self.get_data_from_xml_doc(oval_file, metadata)
  File "/tmp/vulnerablecode/vulnerabilities/data_source.py", line 482, in get_data_from_xml_doc
    raw_data = oval_doc.get_data()
  File "/tmp/vulnerablecode/vulnerabilities/oval_parser.py", line 76, in get_data
    test_data['version_ranges'] = self.get_versionsrngs_from_state(
  File "/tmp/vulnerablecode/vulnerabilities/oval_parser.py", line 152, in get_versionsrngs_from_state
    return RangeSpecifier(version_range)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/dephell_specifier/range_specifier.py", line 40, in __init__
    self._specs = self._parse(spec)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/dephell_specifier/range_specifier.py", line 62, in _parse
    result.add(cls._parse_star_and_operator(constr))
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/dephell_specifier/range_specifier.py", line 123, in _parse_star_and_operator
    parts = version.release[:-1] + (version.release[-1] + 1, )
TypeError: 'NoneType' object is not subscriptable



Failed to get updated_advisories: <xml.etree.ElementTree.ElementTree object at 0x7f5fcdbbe0a0> with {'type': 'deb', 'namespace': 'ubuntu'}:
Traceback (most recent call last):
  File "/tmp/vulnerablecode/vulnerabilities/data_source.py", line 453, in updated_advisories
    oval_data = self.get_data_from_xml_doc(oval_file, metadata)
  File "/tmp/vulnerablecode/vulnerabilities/data_source.py", line 482, in get_data_from_xml_doc
    raw_data = oval_doc.get_data()
  File "/tmp/vulnerablecode/vulnerabilities/oval_parser.py", line 76, in get_data
    test_data['version_ranges'] = self.get_versionsrngs_from_state(
  File "/tmp/vulnerablecode/vulnerabilities/oval_parser.py", line 152, in get_versionsrngs_from_state
    return RangeSpecifier(version_range)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/dephell_specifier/range_specifier.py", line 40, in __init__
    self._specs = self._parse(spec)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/dephell_specifier/range_specifier.py", line 62, in _parse
    result.add(cls._parse_star_and_operator(constr))
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/dephell_specifier/range_specifier.py", line 123, in _parse_star_and_operator
    parts = version.release[:-1] + (version.release[-1] + 1, )
TypeError: 'NoneType' object is not subscriptable





Traceback (most recent call last):
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.ProgramLimitExceeded: string too long to represent as jsonb string
LINE 1: ...ties_importproblem" ("conflicting_model") VALUES ('"[{\"mode...
                                                             ^
DETAIL:  Due to an implementation restriction, jsonb strings cannot exceed 268435455 bytes.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/core/management/base.py", line 328, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/core/management/base.py", line 369, in execute
    output = self.handle(*args, **options)
  File "/tmp/vulnerablecode/vulnerabilities/management/commands/import.py", line 69, in handle
    self._import_data(Importer.objects.all(), options['cutoff_date'])
  File "/tmp/vulnerablecode/vulnerabilities/management/commands/import.py", line 105, in _import_data
    ImportRunner(importer, batch_size).run(cutoff_date=cutoff_date)
  File "/tmp/vulnerablecode/vulnerabilities/import_runner.py", line 115, in run
    process_advisories(data_source)
  File "/tmp/vulnerablecode/vulnerabilities/import_runner.py", line 197, in process_advisories
    handle_conflicts([i.to_model_object() for i in conflicts])
  File "/tmp/vulnerablecode/vulnerabilities/import_runner.py", line 241, in handle_conflicts
    models.ImportProblem.objects.create(conflicting_model=conflicts)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/models/query.py", line 433, in create
    obj.save(force_insert=True, using=self.db)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/models/base.py", line 745, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/models/base.py", line 782, in save_base
    updated = self._save_table(
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/models/base.py", line 887, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/models/base.py", line 924, in _do_insert
    return manager._insert(
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/models/query.py", line 1204, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1392, in execute_sql
    cursor.execute(sql, params)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/backends/utils.py", line 100, in execute
    return super().execute(sql, params)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/tmp/vulnerablecode/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.OperationalError: string too long to represent as jsonb string
LINE 1: ...ties_importproblem" ("conflicting_model") VALUES ('"[{\"mode...
                                                             ^
DETAIL:  Due to an implementation restriction, jsonb strings cannot exceed 268435455 bytes.

[sbs2001]

This is one failing test

In [1]: from  vulnerabilities import models                                                                                                                                                                          

In [2]: from  vulnerabilities.data_source import Advisory                                                                                                                                                            

In [3]: from vulnerabilities.import_runner import _get_or_create_vulnerability                                                                                                                                       

In [4]: adv1 = Advisory(summary="foobar", impacted_package_urls=[],resolved_package_urls=[],cve_id="12")                                                                                                             

In [5]: adv2 = Advisory(summary="foobar", impacted_package_urls=[],resolved_package_urls=[],cve_id="13")                                                                                                             

In [6]: adv3 = Advisory(summary="foobar", impacted_package_urls=[],resolved_package_urls=[],cve_id="")                                                                                                               

In [7]: _get_or_create_vulnerability(adv1)                                                                                                                                                                           
Out[7]: (<Vulnerability: 12>, True)

In [8]: _get_or_create_vulnerability(adv2)                                                                                                                                                                           
Out[8]: (<Vulnerability: 13>, True)

In [9]: _get_or_create_vulnerability(adv3)                                                                                                                                                                           
---------------------------------------------------------------------------
MultipleObjectsReturned                   Traceback (most recent call last)
<ipython-input-9-cde82b468fba> in <module>
----> 1 _get_or_create_vulnerability(adv3)

~/coding/development/vulnerablecode/vulnerabilities/import_runner.py in _get_or_create_vulnerability(advisory)
    250         return models.Vulnerability.objects.create(), True
    251 
--> 252     vuln, created = models.Vulnerability.objects.get_or_create(**query_kwargs)
    253 
    254     # Eventually we only want to keep summary from NVD and ignore other descriptions.

~/coding/development/vulnerablecode/venv/lib/python3.8/site-packages/django/db/models/manager.py in manager_method(self, *args, **kwargs)
     80         def create_method(name, method):
     81             def manager_method(self, *args, **kwargs):
---> 82                 return getattr(self.get_queryset(), name)(*args, **kwargs)
     83             manager_method.__name__ = method.__name__
     84             manager_method.__doc__ = method.__doc__

~/coding/development/vulnerablecode/venv/lib/python3.8/site-packages/django/db/models/query.py in get_or_create(self, defaults, **kwargs)
    557         self._for_write = True
    558         try:
--> 559             return self.get(**kwargs), False
    560         except self.model.DoesNotExist:
    561             params = self._extract_model_params(defaults, **kwargs)

~/coding/development/vulnerablecode/venv/lib/python3.8/site-packages/django/db/models/query.py in get(self, *args, **kwargs)
    417                 self.model._meta.object_name
    418             )
--> 419         raise self.model.MultipleObjectsReturned(
    420             'get() returned more than one %s -- it returned %s!' % (
    421                 self.model._meta.object_name,

MultipleObjectsReturned: get() returned more than one Vulnerability -- it returned 2!

[pombredanne]

This is one failing test

perfect 👍 Let's turn this in a unit test

[pombredanne]

note that in your examples in #266 (comment) I would treat adv1, adv2 and adv3 as three different vulnerabilities for now. The only way to confirm that they may be the same would be on more data (which in our case would be more references)

[sbs2001]

@pombredanne

The test is at https://github.com/sbs2001/vulnerablecode/blob/8717f5f22552821ca170326328c343e6c16aa147/vulnerabilities/tests/test_import_runner.py#L333 .

I've added a crude fix in that branch at https://github.com/sbs2001/vulnerablecode/blob/8717f5f22552821ca170326328c343e6c16aa147/vulnerabilities/import_runner.py#L252

[sbs2001]

The ubuntu oval importer is erroring out due to a bug/feature of dephell_specifier : Here's a test case :

In [3]: z = RangeSpecifier("1:10.1.44-0ubuntu0.18.04.1")                                                                                       

In [4]: z = RangeSpecifier("<1:10.1.44-0ubuntu0.18.04.1")                                                                                      

In [5]: z = RangeSpecifier("<0:8.0")                                                                                                           

In [6]: z = RangeSpecifier("<0:8.0.y")                                                                                                         

In [7]: z = RangeSpecifier("<0:8.0.x")                                                                                                         
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-7-19eff0bdf4ed> in <module>
----> 1 z = RangeSpecifier("<0:8.0.x")

~/coding/opensource/vulnerablecode/venv/lib/python3.8/site-packages/dephell_specifier/range_specifier.py in __init__(self, spec)
     38             return
     39 
---> 40         self._specs = self._parse(spec)
     41         self.join_type = JoinTypes.AND
     42         return

~/coding/opensource/vulnerablecode/venv/lib/python3.8/site-packages/dephell_specifier/range_specifier.py in _parse(cls, spec)
     60             # parse mixed stars and operators like `<=1.2.*`
     61             if constr[0] in '<>' and '.*' in constr:
---> 62                 result.add(cls._parse_star_and_operator(constr))
     63                 continue
     64             # parse npm-style semver specifiers

~/coding/opensource/vulnerablecode/venv/lib/python3.8/site-packages/dephell_specifier/range_specifier.py in _parse_star_and_operator(constr)
    121 
    122         version = parse(constr.lstrip(OPERATOR_SYMBOLS).rstrip('.*'))
--> 123         parts = version.release[:-1] + (version.release[-1] + 1, )
    124         return Specifier(constr[:2] + '.'.join(map(str, parts)))
    125 

TypeError: 'NoneType' object is not subscriptable

having .x in the version range expression is causing the problem.

@pombredanne what would be the optimal workaround here ?

[pombredanne]

@sbs2001 this is a bug upstream IMHO
@orsinium what's your take?

[orsinium]

1. What version specification is it? PEP-440 uses * instead of x, AFAIK.
2. Is it allowed by the spec? Other formats, like NPM semver, don't have a mixture of range and x-range.
3. Feel free to open PR.

[sbs2001]

@orsinium Sorry for the late response,

About

What version specification is it? PEP-440 uses * instead of x, AFAIK.

Does that mean <0:8.0.x is equivalent to <0:8.0.* ?

If yes, then as a quick workaround I was thinking to replace the x with * before creating a RangeSpecifier object .

[sbs2001]

Closing this as mentioned issue is fixed via 2dc2802#diff-88ac39b07d5408fa55869b4c1021fcf3aa97caa4c6a9f16e0f9a4c03bfea1317 and vulcoids