From 8fa6eeb60a834ee10fb384178a44bae598e94447 Mon Sep 17 00:00:00 2001 From: alessiobuscemi Date: Thu, 27 Nov 2025 12:06:19 +0100 Subject: [PATCH 01/16] Create citcom_label.md --- docs/toolbox/citcom_label.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 docs/toolbox/citcom_label.md diff --git a/docs/toolbox/citcom_label.md b/docs/toolbox/citcom_label.md new file mode 100644 index 00000000..8077c596 --- /dev/null +++ b/docs/toolbox/citcom_label.md @@ -0,0 +1,28 @@ +# Citcom Label + +The Citcom Label is an initiative currently under development within Citcom.ai. Its goal is to create a trusted, recognisable signal that helps AI providers demonstrate responsible practices and gives buyers—especially public-sector actors such as smart cities—a clearer basis for evaluating and procuring AI solutions. + +## What will the Citcom Label be? + +The label is envisioned as a **system of digital badges**, each representing a specific dimension of trustworthiness assessed during the evaluation process. +These badges would include a **watermark**, ensuring authenticity and preventing misuse. Each badge would be **verifiable through the Citcom Hub**, allowing external stakeholders to confirm its origin, evaluation status, and associated criteria. + +The Citcom badges are **not intended to function as legally binding conformity certificates under the AI Act**. Instead, they serve as **smart-city–oriented quality marks**, helping cities and other public authorities gain confidence in the AI solutions they consider adopting. + +For AI innovators, the Citcom badge system provides **independent third-party validation**, helping them promote their solutions and demonstrate that they meet recognised standards of trustworthiness. For cities and public buyers, the badges offer **clear, evidence-based guidance** to support more informed and transparent procurement decisions. + +## On what basis will the Citcom badges be awarded? + +The detailed criteria are still being developed with Citcom partners, but several guiding principles are emerging: + +### Completion of an evaluation +A badge is expected to be awarded only once a solution completes a structured assessment aligned with shared guidelines for the relevant dimension of trustworthiness. + +### Common methodology +Work is ongoing to define a coherent framework that determines how systems are qualified, how requirements translate into test cases, and how results are interpreted across different trust dimensions. + +### Success thresholds +Initial discussions point toward setting minimum quantitative and qualitative thresholds that vary by product type, maturity level, and the specific dimension being assessed. + +### Real-world validation +Evaluations are expected to rely on practical or pilot scenarios using the actual product, ensuring that results reflect real-world behaviour. From 3e3e289cac78aef7f9c08541249802d2b04c76f2 Mon Sep 17 00:00:00 2001 From: alessiobuscemi Date: Thu, 27 Nov 2025 13:08:15 +0100 Subject: [PATCH 02/16] Update citcom_label.md --- docs/toolbox/citcom_label.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/toolbox/citcom_label.md b/docs/toolbox/citcom_label.md index 8077c596..d6c32386 100644 --- a/docs/toolbox/citcom_label.md +++ b/docs/toolbox/citcom_label.md @@ -2,6 +2,7 @@ The Citcom Label is an initiative currently under development within Citcom.ai. Its goal is to create a trusted, recognisable signal that helps AI providers demonstrate responsible practices and gives buyers—especially public-sector actors such as smart cities—a clearer basis for evaluating and procuring AI solutions. + ## What will the Citcom Label be? The label is envisioned as a **system of digital badges**, each representing a specific dimension of trustworthiness assessed during the evaluation process. @@ -26,3 +27,26 @@ Initial discussions point toward setting minimum quantitative and qualitative th ### Real-world validation Evaluations are expected to rely on practical or pilot scenarios using the actual product, ensuring that results reflect real-world behaviour. + + +## Who will conduct the assessment and with which methodologies? + +The assessment behind each Citcom badge will be carried out by the participating TEF sites. Each site brings its own specialised methodologies, tools, and testing infrastructures, reflecting the diversity of technical expertise across the Citcom network. + +These assessment solutions cover different dimensions of trustworthiness and can be consulted through the **AI Assessment Catalogue**, available at the following link: + +**** + +The catalogue provides an overview of the available evaluation tools, test suites, and methodologies, enabling innovators to understand which capabilities are applied to their systems and helping cities see how specific trust dimensions are assessed. + +### Can an AI provider receive assessments across multiple TEF sites? + +Yes. If a solution would benefit from complementary expertise available across several TEF sites, an AI provider can undergo assessments in multiple locations. In such cases, the **first-contact TEF site** will coordinate the overall process. + +The coordinating TEF site will: +- connect with the additional TEF sites that carry out their assessments independently, +- ensure that each participating site manages its own contractual and operational responsibilities, +- consolidate the evaluation results into a unified report, +- and oversee the issuance of the Citcom badges corresponding to the dimensions assessed across all sites. + +This ensures a seamless experience for AI innovators while leveraging the full breadth of expertise across the TEF network. From f3387af1ce4648064ea784841bcc8bcae2a0d3d6 Mon Sep 17 00:00:00 2001 From: alessiobuscemi Date: Thu, 27 Nov 2025 13:09:43 +0100 Subject: [PATCH 03/16] Create citcom_label.md --- docs/ai_assessment_catalog/citcom_label.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/ai_assessment_catalog/citcom_label.md diff --git a/docs/ai_assessment_catalog/citcom_label.md b/docs/ai_assessment_catalog/citcom_label.md new file mode 100644 index 00000000..8b137891 --- /dev/null +++ b/docs/ai_assessment_catalog/citcom_label.md @@ -0,0 +1 @@ + From 82eb1737a4eaf252484357e05b74b3ea7885311b Mon Sep 17 00:00:00 2001 From: alessiobuscemi Date: Thu, 27 Nov 2025 13:10:00 +0100 Subject: [PATCH 04/16] Update citcom_label.md --- docs/ai_assessment_catalog/citcom_label.md | 51 ++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/docs/ai_assessment_catalog/citcom_label.md b/docs/ai_assessment_catalog/citcom_label.md index 8b137891..d6c32386 100644 --- a/docs/ai_assessment_catalog/citcom_label.md +++ b/docs/ai_assessment_catalog/citcom_label.md @@ -1 +1,52 @@ +# Citcom Label +The Citcom Label is an initiative currently under development within Citcom.ai. Its goal is to create a trusted, recognisable signal that helps AI providers demonstrate responsible practices and gives buyers—especially public-sector actors such as smart cities—a clearer basis for evaluating and procuring AI solutions. + + +## What will the Citcom Label be? + +The label is envisioned as a **system of digital badges**, each representing a specific dimension of trustworthiness assessed during the evaluation process. +These badges would include a **watermark**, ensuring authenticity and preventing misuse. Each badge would be **verifiable through the Citcom Hub**, allowing external stakeholders to confirm its origin, evaluation status, and associated criteria. + +The Citcom badges are **not intended to function as legally binding conformity certificates under the AI Act**. Instead, they serve as **smart-city–oriented quality marks**, helping cities and other public authorities gain confidence in the AI solutions they consider adopting. + +For AI innovators, the Citcom badge system provides **independent third-party validation**, helping them promote their solutions and demonstrate that they meet recognised standards of trustworthiness. For cities and public buyers, the badges offer **clear, evidence-based guidance** to support more informed and transparent procurement decisions. + +## On what basis will the Citcom badges be awarded? + +The detailed criteria are still being developed with Citcom partners, but several guiding principles are emerging: + +### Completion of an evaluation +A badge is expected to be awarded only once a solution completes a structured assessment aligned with shared guidelines for the relevant dimension of trustworthiness. + +### Common methodology +Work is ongoing to define a coherent framework that determines how systems are qualified, how requirements translate into test cases, and how results are interpreted across different trust dimensions. + +### Success thresholds +Initial discussions point toward setting minimum quantitative and qualitative thresholds that vary by product type, maturity level, and the specific dimension being assessed. + +### Real-world validation +Evaluations are expected to rely on practical or pilot scenarios using the actual product, ensuring that results reflect real-world behaviour. + + +## Who will conduct the assessment and with which methodologies? + +The assessment behind each Citcom badge will be carried out by the participating TEF sites. Each site brings its own specialised methodologies, tools, and testing infrastructures, reflecting the diversity of technical expertise across the Citcom network. + +These assessment solutions cover different dimensions of trustworthiness and can be consulted through the **AI Assessment Catalogue**, available at the following link: + +**** + +The catalogue provides an overview of the available evaluation tools, test suites, and methodologies, enabling innovators to understand which capabilities are applied to their systems and helping cities see how specific trust dimensions are assessed. + +### Can an AI provider receive assessments across multiple TEF sites? + +Yes. If a solution would benefit from complementary expertise available across several TEF sites, an AI provider can undergo assessments in multiple locations. In such cases, the **first-contact TEF site** will coordinate the overall process. + +The coordinating TEF site will: +- connect with the additional TEF sites that carry out their assessments independently, +- ensure that each participating site manages its own contractual and operational responsibilities, +- consolidate the evaluation results into a unified report, +- and oversee the issuance of the Citcom badges corresponding to the dimensions assessed across all sites. + +This ensures a seamless experience for AI innovators while leveraging the full breadth of expertise across the TEF network. From 85c24da335993b43d10d57f3827ccfc412c89506 Mon Sep 17 00:00:00 2001 From: alessiobuscemi Date: Thu, 27 Nov 2025 13:12:15 +0100 Subject: [PATCH 05/16] Update index.md --- docs/ai_assessment_catalog/index.md | 100 +++++++++++++++------------- 1 file changed, 52 insertions(+), 48 deletions(-) diff --git a/docs/ai_assessment_catalog/index.md b/docs/ai_assessment_catalog/index.md index f7e07014..d6c32386 100644 --- a/docs/ai_assessment_catalog/index.md +++ b/docs/ai_assessment_catalog/index.md @@ -1,48 +1,52 @@ ---- -icon: material/store-search-outline -title: AI Assessment Catalog -hide: - - toc ---- - - - -
-
-[_PAGE DESCRIPTION_] - -
-
- - -
-
- - -
- -
- -| Dataset | Super Node | TEF Node | Site | Data Model | Sampling Time | Historical | Owner | Get Access | -| ------- | ---------- | -------- | ---- | ---------- | ------------- | ---------- | ----- | ---------- | -| [Waste Container](./metadata_datasets/south_spain_valencia.md) | South | Spain | Valencia | [gitlab_vlci](https://gitlab.com/vlci-public/models-dades/wastemanagement/-/blob/main/WasteContainer/spec.md?ref_type=heads) | RealTime | From 2000 | València City Council | [Contact](https://valencia.opendatasoft.com/pages/home/) | +# Citcom Label + +The Citcom Label is an initiative currently under development within Citcom.ai. Its goal is to create a trusted, recognisable signal that helps AI providers demonstrate responsible practices and gives buyers—especially public-sector actors such as smart cities—a clearer basis for evaluating and procuring AI solutions. + + +## What will the Citcom Label be? + +The label is envisioned as a **system of digital badges**, each representing a specific dimension of trustworthiness assessed during the evaluation process. +These badges would include a **watermark**, ensuring authenticity and preventing misuse. Each badge would be **verifiable through the Citcom Hub**, allowing external stakeholders to confirm its origin, evaluation status, and associated criteria. + +The Citcom badges are **not intended to function as legally binding conformity certificates under the AI Act**. Instead, they serve as **smart-city–oriented quality marks**, helping cities and other public authorities gain confidence in the AI solutions they consider adopting. + +For AI innovators, the Citcom badge system provides **independent third-party validation**, helping them promote their solutions and demonstrate that they meet recognised standards of trustworthiness. For cities and public buyers, the badges offer **clear, evidence-based guidance** to support more informed and transparent procurement decisions. + +## On what basis will the Citcom badges be awarded? + +The detailed criteria are still being developed with Citcom partners, but several guiding principles are emerging: + +### Completion of an evaluation +A badge is expected to be awarded only once a solution completes a structured assessment aligned with shared guidelines for the relevant dimension of trustworthiness. + +### Common methodology +Work is ongoing to define a coherent framework that determines how systems are qualified, how requirements translate into test cases, and how results are interpreted across different trust dimensions. + +### Success thresholds +Initial discussions point toward setting minimum quantitative and qualitative thresholds that vary by product type, maturity level, and the specific dimension being assessed. + +### Real-world validation +Evaluations are expected to rely on practical or pilot scenarios using the actual product, ensuring that results reflect real-world behaviour. + + +## Who will conduct the assessment and with which methodologies? + +The assessment behind each Citcom badge will be carried out by the participating TEF sites. Each site brings its own specialised methodologies, tools, and testing infrastructures, reflecting the diversity of technical expertise across the Citcom network. + +These assessment solutions cover different dimensions of trustworthiness and can be consulted through the **AI Assessment Catalogue**, available at the following link: + +**** + +The catalogue provides an overview of the available evaluation tools, test suites, and methodologies, enabling innovators to understand which capabilities are applied to their systems and helping cities see how specific trust dimensions are assessed. + +### Can an AI provider receive assessments across multiple TEF sites? + +Yes. If a solution would benefit from complementary expertise available across several TEF sites, an AI provider can undergo assessments in multiple locations. In such cases, the **first-contact TEF site** will coordinate the overall process. + +The coordinating TEF site will: +- connect with the additional TEF sites that carry out their assessments independently, +- ensure that each participating site manages its own contractual and operational responsibilities, +- consolidate the evaluation results into a unified report, +- and oversee the issuance of the Citcom badges corresponding to the dimensions assessed across all sites. + +This ensures a seamless experience for AI innovators while leveraging the full breadth of expertise across the TEF network. From 728326632f17bef74be35f5e6cde1b8304920c3f Mon Sep 17 00:00:00 2001 From: alessio0208 Date: Thu, 27 Nov 2025 13:41:59 +0100 Subject: [PATCH 06/16] added citcom label and ai_assessment_catalogue --- docs/ai_assessment_catalog/citcom_label.md | 52 ------------------- docs/citcom_label/ai_assessment_catalogue.md | 50 ++++++++++++++++++ .../index.md | 2 +- docs/data_catalog/index.md | 36 ++++++------- 4 files changed, 66 insertions(+), 74 deletions(-) delete mode 100644 docs/ai_assessment_catalog/citcom_label.md create mode 100644 docs/citcom_label/ai_assessment_catalogue.md rename docs/{ai_assessment_catalog => citcom_label}/index.md (98%) diff --git a/docs/ai_assessment_catalog/citcom_label.md b/docs/ai_assessment_catalog/citcom_label.md deleted file mode 100644 index d6c32386..00000000 --- a/docs/ai_assessment_catalog/citcom_label.md +++ /dev/null @@ -1,52 +0,0 @@ -# Citcom Label - -The Citcom Label is an initiative currently under development within Citcom.ai. Its goal is to create a trusted, recognisable signal that helps AI providers demonstrate responsible practices and gives buyers—especially public-sector actors such as smart cities—a clearer basis for evaluating and procuring AI solutions. - - -## What will the Citcom Label be? - -The label is envisioned as a **system of digital badges**, each representing a specific dimension of trustworthiness assessed during the evaluation process. -These badges would include a **watermark**, ensuring authenticity and preventing misuse. Each badge would be **verifiable through the Citcom Hub**, allowing external stakeholders to confirm its origin, evaluation status, and associated criteria. - -The Citcom badges are **not intended to function as legally binding conformity certificates under the AI Act**. Instead, they serve as **smart-city–oriented quality marks**, helping cities and other public authorities gain confidence in the AI solutions they consider adopting. - -For AI innovators, the Citcom badge system provides **independent third-party validation**, helping them promote their solutions and demonstrate that they meet recognised standards of trustworthiness. For cities and public buyers, the badges offer **clear, evidence-based guidance** to support more informed and transparent procurement decisions. - -## On what basis will the Citcom badges be awarded? - -The detailed criteria are still being developed with Citcom partners, but several guiding principles are emerging: - -### Completion of an evaluation -A badge is expected to be awarded only once a solution completes a structured assessment aligned with shared guidelines for the relevant dimension of trustworthiness. - -### Common methodology -Work is ongoing to define a coherent framework that determines how systems are qualified, how requirements translate into test cases, and how results are interpreted across different trust dimensions. - -### Success thresholds -Initial discussions point toward setting minimum quantitative and qualitative thresholds that vary by product type, maturity level, and the specific dimension being assessed. - -### Real-world validation -Evaluations are expected to rely on practical or pilot scenarios using the actual product, ensuring that results reflect real-world behaviour. - - -## Who will conduct the assessment and with which methodologies? - -The assessment behind each Citcom badge will be carried out by the participating TEF sites. Each site brings its own specialised methodologies, tools, and testing infrastructures, reflecting the diversity of technical expertise across the Citcom network. - -These assessment solutions cover different dimensions of trustworthiness and can be consulted through the **AI Assessment Catalogue**, available at the following link: - -**** - -The catalogue provides an overview of the available evaluation tools, test suites, and methodologies, enabling innovators to understand which capabilities are applied to their systems and helping cities see how specific trust dimensions are assessed. - -### Can an AI provider receive assessments across multiple TEF sites? - -Yes. If a solution would benefit from complementary expertise available across several TEF sites, an AI provider can undergo assessments in multiple locations. In such cases, the **first-contact TEF site** will coordinate the overall process. - -The coordinating TEF site will: -- connect with the additional TEF sites that carry out their assessments independently, -- ensure that each participating site manages its own contractual and operational responsibilities, -- consolidate the evaluation results into a unified report, -- and oversee the issuance of the Citcom badges corresponding to the dimensions assessed across all sites. - -This ensures a seamless experience for AI innovators while leveraging the full breadth of expertise across the TEF network. diff --git a/docs/citcom_label/ai_assessment_catalogue.md b/docs/citcom_label/ai_assessment_catalogue.md new file mode 100644 index 00000000..37304163 --- /dev/null +++ b/docs/citcom_label/ai_assessment_catalogue.md @@ -0,0 +1,50 @@ +--- +icon: material/robot-search-outline +title: AI Assessment Catalogue +hide: + - toc +--- + + + +
+
+The AI Assessment Catalogue is a centralized hub showcasing the evaluation tools, testing frameworks, and assessment solutions available across the Citcom.ai TEF network. +It is regularly updated as new methodologies and tools become available at each TEF site. +If you would like to request an assessment or learn more about a tool, please contact the relevant TEF sites. + + +
+
+ + +
+ +
+ +| Solution Name | Provider | Licensing Type | Project Phase / TRL | Domain of Application | AI Risk Category | Ethical Dimensions | Security & Securitization of Data | Example of Use Case | Resources | +|---------------|----------|----------------|----------------------|------------------------|------------------|--------------------|-----------------------------------|----------------------|-----------| +| **FAIRGAME** | LIST | Open-source | TRL 6–8 | LLM bias testing, AI agents behavioural testing, jailbreaking testing | General Purpose AI | Fairness, Robustness | Depends on the use case (whether the chatbot/AI agent has access to sensitive data) | A city aims to test its citizen-facing chatbot before launch. FAIRGAME enables the creation of simulated users with diverse identities, personalities, and requests using LLMs, allowing evaluation in dynamic, real-world-like conversations. | GitHub: , Paper: | +| **MLA-BiTe** | LIST | To be open sourced | TRL 6–8 | LLM bias testing | General Purpose AI | Fairness, Robustness | No data privacy requirements | A city plans to evaluate fairness in its citizen-facing chatbot. MLA-BiTe allows non-technical staff to create local scenario-based prompts to uncover discriminatory behaviour across sensitive categories, supporting multiple languages and augmentations. | — | +| **Legal KG-RAG** | LIST | Proprietary | TRL 5–7 | LLM factuality accuracy testing | General Purpose AI | Transparency, Explainability, Robustness | Depends on whether the RAG is performed on sensitive data | A city using a standard RAG pipeline obtains irrelevant results. Legal KG-RAG rebuilds the legal corpus as a Neo4j knowledge graph, enabling direct comparison between traditional and KG-enhanced retrieval. | — | +| **MLA-Reject** | LIST | To be open sourced | TRL 6–8 | LLM robustness to jailbreaking | General Purpose AI | Robustness | Depends on whether the system has access to sensitive data | A public administration operates a multilingual assistant for internal queries. They want to test robustness against unsafe or misleading prompts. MLA-Reject generates difficult negative prompts to test refusal behaviour and safety guardrails, revealing weaknesses and improving configurations. | — | diff --git a/docs/ai_assessment_catalog/index.md b/docs/citcom_label/index.md similarity index 98% rename from docs/ai_assessment_catalog/index.md rename to docs/citcom_label/index.md index d6c32386..7f6818b3 100644 --- a/docs/ai_assessment_catalog/index.md +++ b/docs/citcom_label/index.md @@ -35,7 +35,7 @@ The assessment behind each Citcom badge will be carried out by the participating These assessment solutions cover different dimensions of trustworthiness and can be consulted through the **AI Assessment Catalogue**, available at the following link: -**** +[AI Assessment Catalogue](ai_assessment_catalogue.md) The catalogue provides an overview of the available evaluation tools, test suites, and methodologies, enabling innovators to understand which capabilities are applied to their systems and helping cities see how specific trust dimensions are assessed. diff --git a/docs/data_catalog/index.md b/docs/data_catalog/index.md index d6fd4bd3..bc2f2611 100644 --- a/docs/data_catalog/index.md +++ b/docs/data_catalog/index.md @@ -1,12 +1,12 @@ --- -icon: material/store-search-outline -title: Data Catalog +icon: material/robot-search-outline +title: AI Assessment Catalogue hide: - toc --- -
-The data catalog is a centralized hub to keep track of available datasets. It is regularly updated to include new data as it becomes available in any TEF node. If you want access to any dataset, please click "Contact" to reach the owners. +The AI Assessment Catalogue is a centralized hub showcasing the evaluation tools, testing frameworks, and assessment solutions available across the TEF network. +It is regularly updated as new methodologies and tools become available at each TEF site. +If you would like to request an assessment or learn more about a tool, please click **Contact** or open the linked resources. -!!! question "[How to add new datasets?](./instructions.md)" +!!! question "[How to add new assessment solutions?](./instructions.md)"
-[:simple-github: Add New Datasets ](https://github.com/CitComAI-Hub/CitComAI-Hub.github.io/issues/new/choose){:target="_blank" .md-button .md-button--primary-light } +[:simple-github: Add New Assessment Solution](https://github.com/CitComAI-Hub/CitComAI-Hub.github.io/issues/new/choose){:target="_blank" .md-button .md-button--primary-light }
@@ -35,7 +36,7 @@ The data catalog is a centralized hub to keep track of available datasets. It is - +
-| Dataset | Super Node | TEF Node | Site | Data Model | Sampling Time | Historical | Owner | Get Access | -| ------- | ---------- | -------- | ---- | ---------- | ------------- | ---------- | ----- | ---------- | -| [Waste Container](./metadata_datasets/south_spain_valencia.md) | South | Spain | Valencia | [gitlab_vlci](https://gitlab.com/vlci-public/models-dades/wastemanagement/-/blob/main/WasteContainer/spec.md?ref_type=heads) | RealTime | From 2000 | València City Council | [Contact](https://valencia.opendatasoft.com/pages/home/) | -| [Weather Forecast](./metadata_datasets/south_spain_valencia.md) | South | Spain | Valencia | [gitlab_vlci](https://gitlab.com/vlci-public/models-dades/weather/blob/main/WeatherForecast/spec.md) | Daily | From 2010 | València City Council | [Contact](https://valencia.opendatasoft.com/pages/home/) | -| [Bikeparking stands in Aarhus City](./metadata_datasets/nordic_citcom_gtm.md) | Nordic | Denmark | GTM | *no specific value* | Ongoing | No | Aarhus Municipality | [City of Aarhus](https://www.opendata.dk/city-of-aarhus/cykelparkering_aarhus) | -| [Bike terminals in Aarhus, air and tools](./metadata_datasets/nordic_citcom_gtm.md) | Nordic | Denmark | GTM | *no specific value* | Ongoing | No | Aarhus Municipality | [Cykelterminal - Dataset](https://www.opendata.dk/city-of-aarhus/cykelterminal) | -| [Citybike locations in Aarhus](./metadata_datasets/nordic_citcom_gtm.md) | Nordic | Denmark | GTM | *no specific value* | *no specific value* | No | Aarhus Municipality | [Aarhus Bycykel - Dataset](https://www.opendata.dk/city-of-aarhus/aarhus-bycykel) | -| [Fast track bikeroutes in Aarhus](./metadata_datasets/nordic_citcom_gtm.md) | Nordic | Denmark | GTM | *no specific value* | Ongoing | No | Aarhus Municipality | [Supercykelsti i Aarhus Kommune - Dataset](https://www.opendata.dk/city-of-aarhus/supercykelsti) | -| [recreative bikeroutes in Aarhus](./metadata_datasets/nordic_citcom_gtm.md) | Nordic | Denmark | GTM | *no specific value* | Ongoing | No | Aarhus Municipality | [Rekreative cykelruter - Dataset](https://www.opendata.dk/city-of-aarhus/rekreative-cykelruter) | -| [AirQuality](./metadata_datasets/south_italy_uptown.md) | South | Italy | UpTown | *no specific value* | 30s | Yes | Politecnico di Milano | [neslab.it](https://www.neslab.it) | -| [Biodiversity](./metadata_datasets/south_italy_uptown.md) | South | Italy | UpTown | *no specific value* | 15m | Yes | Politecnico di Milano | [neslab.it](https://www.neslab.it) | -| [Energy distribution](./metadata_datasets/south_italy_uptown.md) | South | Italy | UpTown | *no specific value* | Hourly | Yes | A2A | [neslab.it](https://www.neslab.it) | -| [Archaeological Site](./metadata_datasets/south_italy_mithraeum-of-circus-maximus.md) | South | Italy | Mithraeum of Circus Maximus | *no specific value* | Real-time | Yes | Politecnico di Milano | [neslab.it](https://www.neslab.it) | +| Solution Name | Provider | Licensing Type | Project Phase / TRL | Domain of Application | AI Risk Category | Ethical Dimensions | Security & Securitization of Data | Example of Use Case | Resources | +|---------------|----------|----------------|----------------------|------------------------|------------------|--------------------|-----------------------------------|----------------------|-----------| +| **FAIRGAME** | LIST | Open-source | TRL 6–8 | LLM bias testing, AI agents behavioural testing, jailbreaking testing | General Purpose AI | Fairness, Robustness | Depends on the use case (whether the chatbot/AI agent has access to sensitive data) | A city aims to test its citizen-facing chatbot before launch. FAIRGAME enables the creation of simulated users with diverse identities, personalities, and requests using LLMs, allowing evaluation in dynamic, real-world-like conversations. | GitHub: , Paper: | +| **MLA-BiTe** | LIST | To be open sourced | TRL 6–8 | LLM bias testing | General Purpose AI | Fairness, Robustness | No data privacy requirements | A city plans to evaluate fairness in its citizen-facing chatbot. MLA-BiTe allows non-technical staff to create local scenario-based prompts to uncover discriminatory behaviour across sensitive categories, supporting multiple languages and augmentations. | — | +| **Legal KG-RAG** | LIST | Proprietary | TRL 5–7 | LLM factuality accuracy testing | General Purpose AI | Transparency, Explainability, Robustness | Depends on whether the RAG is performed on sensitive data | A city using a standard RAG pipeline obtains irrelevant results. Legal KG-RAG rebuilds the legal corpus as a Neo4j knowledge graph, enabling direct comparison between traditional and KG-enhanced retrieval. | — | +| **MLA-Reject** | LIST | To be open sourced | TRL 6–8 | LLM robustness to jailbreaking | General Purpose AI | Robustness | Depends on whether the system has access to sensitive data | A public administration operates a multilingual assistant for internal queries. They want to test robustness against unsafe or misleading prompts. MLA-Reject generates difficult negative prompts to test refusal behaviour and safety guardrails, revealing weaknesses and improving configurations. | — | From 29e9575e9b8e68c2a1f2ad118333808bff00d980 Mon Sep 17 00:00:00 2001 From: alessio0208 Date: Thu, 27 Nov 2025 13:59:40 +0100 Subject: [PATCH 07/16] added citcom label and ai_assessment_catalogue --- docs/citcom_label/ai_assessment_catalogue.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/docs/citcom_label/ai_assessment_catalogue.md b/docs/citcom_label/ai_assessment_catalogue.md index 37304163..de284aab 100644 --- a/docs/citcom_label/ai_assessment_catalogue.md +++ b/docs/citcom_label/ai_assessment_catalogue.md @@ -20,7 +20,7 @@ hide:
-The AI Assessment Catalogue is a centralized hub showcasing the evaluation tools, testing frameworks, and assessment solutions available across the Citcom.ai TEF network. +The AI Assessment Catalogue showcases the evaluation tools, testing frameworks, and assessment solutions available across the Citcom.ai TEF network. It is regularly updated as new methodologies and tools become available at each TEF site. If you would like to request an assessment or learn more about a tool, please contact the relevant TEF sites. @@ -42,6 +42,20 @@ If you would like to request an assessment or learn more about a tool, please co
+ + | Solution Name | Provider | Licensing Type | Project Phase / TRL | Domain of Application | AI Risk Category | Ethical Dimensions | Security & Securitization of Data | Example of Use Case | Resources | |---------------|----------|----------------|----------------------|------------------------|------------------|--------------------|-----------------------------------|----------------------|-----------| | **FAIRGAME** | LIST | Open-source | TRL 6–8 | LLM bias testing, AI agents behavioural testing, jailbreaking testing | General Purpose AI | Fairness, Robustness | Depends on the use case (whether the chatbot/AI agent has access to sensitive data) | A city aims to test its citizen-facing chatbot before launch. FAIRGAME enables the creation of simulated users with diverse identities, personalities, and requests using LLMs, allowing evaluation in dynamic, real-world-like conversations. | GitHub: , Paper: | From 11f64d0735a43199ec8cba5e13f57be0d9843b69 Mon Sep 17 00:00:00 2001 From: alessio0208 Date: Thu, 27 Nov 2025 14:01:19 +0100 Subject: [PATCH 08/16] added citcom label and ai_assessment_catalogue --- docs/citcom_label/ai_assessment_catalogue.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/docs/citcom_label/ai_assessment_catalogue.md b/docs/citcom_label/ai_assessment_catalogue.md index de284aab..2c8cd2b1 100644 --- a/docs/citcom_label/ai_assessment_catalogue.md +++ b/docs/citcom_label/ai_assessment_catalogue.md @@ -43,19 +43,28 @@ If you would like to request an assessment or learn more about a tool, please co
+ | Solution Name | Provider | Licensing Type | Project Phase / TRL | Domain of Application | AI Risk Category | Ethical Dimensions | Security & Securitization of Data | Example of Use Case | Resources | |---------------|----------|----------------|----------------------|------------------------|------------------|--------------------|-----------------------------------|----------------------|-----------| | **FAIRGAME** | LIST | Open-source | TRL 6–8 | LLM bias testing, AI agents behavioural testing, jailbreaking testing | General Purpose AI | Fairness, Robustness | Depends on the use case (whether the chatbot/AI agent has access to sensitive data) | A city aims to test its citizen-facing chatbot before launch. FAIRGAME enables the creation of simulated users with diverse identities, personalities, and requests using LLMs, allowing evaluation in dynamic, real-world-like conversations. | GitHub: , Paper: | From 640f564063d8b08fe17b9334725cb984c56bf7ee Mon Sep 17 00:00:00 2001 From: alessiobuscemi Date: Thu, 27 Nov 2025 14:02:23 +0100 Subject: [PATCH 09/16] Update ai_assessment_catalogue.md --- docs/citcom_label/ai_assessment_catalogue.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/citcom_label/ai_assessment_catalogue.md b/docs/citcom_label/ai_assessment_catalogue.md index 2c8cd2b1..8cf0c44e 100644 --- a/docs/citcom_label/ai_assessment_catalogue.md +++ b/docs/citcom_label/ai_assessment_catalogue.md @@ -65,9 +65,9 @@ If you would like to request an assessment or learn more about a tool, please co -| Solution Name | Provider | Licensing Type | Project Phase / TRL | Domain of Application | AI Risk Category | Ethical Dimensions | Security & Securitization of Data | Example of Use Case | Resources | -|---------------|----------|----------------|----------------------|------------------------|------------------|--------------------|-----------------------------------|----------------------|-----------| -| **FAIRGAME** | LIST | Open-source | TRL 6–8 | LLM bias testing, AI agents behavioural testing, jailbreaking testing | General Purpose AI | Fairness, Robustness | Depends on the use case (whether the chatbot/AI agent has access to sensitive data) | A city aims to test its citizen-facing chatbot before launch. FAIRGAME enables the creation of simulated users with diverse identities, personalities, and requests using LLMs, allowing evaluation in dynamic, real-world-like conversations. | GitHub: , Paper: | -| **MLA-BiTe** | LIST | To be open sourced | TRL 6–8 | LLM bias testing | General Purpose AI | Fairness, Robustness | No data privacy requirements | A city plans to evaluate fairness in its citizen-facing chatbot. MLA-BiTe allows non-technical staff to create local scenario-based prompts to uncover discriminatory behaviour across sensitive categories, supporting multiple languages and augmentations. | — | -| **Legal KG-RAG** | LIST | Proprietary | TRL 5–7 | LLM factuality accuracy testing | General Purpose AI | Transparency, Explainability, Robustness | Depends on whether the RAG is performed on sensitive data | A city using a standard RAG pipeline obtains irrelevant results. Legal KG-RAG rebuilds the legal corpus as a Neo4j knowledge graph, enabling direct comparison between traditional and KG-enhanced retrieval. | — | -| **MLA-Reject** | LIST | To be open sourced | TRL 6–8 | LLM robustness to jailbreaking | General Purpose AI | Robustness | Depends on whether the system has access to sensitive data | A public administration operates a multilingual assistant for internal queries. They want to test robustness against unsafe or misleading prompts. MLA-Reject generates difficult negative prompts to test refusal behaviour and safety guardrails, revealing weaknesses and improving configurations. | — | +| Solution Name | Provider | Licensing Type | Project Phase / TRL | Domain of Application | AI Risk Category | Ethical Dimensions | Security & Securitization of Data | Resources | Example of Use Case | +|---------------|----------|----------------|----------------------|------------------------|------------------|--------------------|-----------------------------------|-----------|----------------------| +| **FAIRGAME** | LIST | Open-source | TRL 6–8 | LLM bias testing, AI agents behavioural testing, jailbreaking testing | General Purpose AI | Fairness, Robustness | Depends on the use case (whether the chatbot/AI agent has access to sensitive data) | GitHub: , Paper: | A city aims to test its citizen-facing chatbot before launch. FAIRGAME enables the creation of simulated users with diverse identities, personalities, and requests using LLMs, allowing evaluation in dynamic, real-world-like conversations. | +| **MLA-BiTe** | LIST | To be open sourced | TRL 6–8 | LLM bias testing | General Purpose AI | Fairness, Robustness | No data privacy requirements | — | A city plans to evaluate fairness in its citizen-facing chatbot. MLA-BiTe allows non-technical staff to create local scenario-based prompts to uncover discriminatory behaviour across sensitive categories, supporting multiple languages and augmentations. | +| **Legal KG-RAG** | LIST | Proprietary | TRL 5–7 | LLM factuality accuracy testing | General Purpose AI | Transparency, Explainability, Robustness | Depends on whether the RAG is performed on sensitive data | — | A city using a standard RAG pipeline obtains irrelevant results. Legal KG-RAG rebuilds the legal corpus as a Neo4j knowledge graph, enabling direct comparison between traditional and KG-enhanced retrieval. | +| **MLA-Reject** | LIST | To be open sourced | TRL 6–8 | LLM robustness to jailbreaking | General Purpose AI | Robustness | Depends on whether the system has access to sensitive data | — | A public administration operates a multilingual assistant for internal queries. They want to test robustness against unsafe or misleading prompts. MLA-Reject generates difficult negative prompts to test refusal behaviour and safety guardrails, revealing weaknesses and improving configurations. | From e6410cedbd30622ecfe746bebeeb587f75eadc12 Mon Sep 17 00:00:00 2001 From: alessiobuscemi Date: Thu, 27 Nov 2025 14:02:55 +0100 Subject: [PATCH 10/16] Update ai_assessment_catalogue.md --- docs/citcom_label/ai_assessment_catalogue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/citcom_label/ai_assessment_catalogue.md b/docs/citcom_label/ai_assessment_catalogue.md index 8cf0c44e..031f5389 100644 --- a/docs/citcom_label/ai_assessment_catalogue.md +++ b/docs/citcom_label/ai_assessment_catalogue.md @@ -67,7 +67,7 @@ If you would like to request an assessment or learn more about a tool, please co | Solution Name | Provider | Licensing Type | Project Phase / TRL | Domain of Application | AI Risk Category | Ethical Dimensions | Security & Securitization of Data | Resources | Example of Use Case | |---------------|----------|----------------|----------------------|------------------------|------------------|--------------------|-----------------------------------|-----------|----------------------| -| **FAIRGAME** | LIST | Open-source | TRL 6–8 | LLM bias testing, AI agents behavioural testing, jailbreaking testing | General Purpose AI | Fairness, Robustness | Depends on the use case (whether the chatbot/AI agent has access to sensitive data) | GitHub: , Paper: | A city aims to test its citizen-facing chatbot before launch. FAIRGAME enables the creation of simulated users with diverse identities, personalities, and requests using LLMs, allowing evaluation in dynamic, real-world-like conversations. | +| **FAIRGAME** | LIST | Open-source | TRL 6–8 | LLM bias testing, AI agents behavioural testing, jailbreaking testing | General Purpose AI | Fairness, Robustness | Depends on the use case (whether the chatbot/AI agent has access to sensitive data) | GitHub: , Paper: | A city aims to test its citizen-facing chatbot before launch. FAIRGAME enables the creation of simulated users with diverse identities, personalities, and requests using LLMs, allowing evaluation in dynamic, real-world-like conversations. | | **MLA-BiTe** | LIST | To be open sourced | TRL 6–8 | LLM bias testing | General Purpose AI | Fairness, Robustness | No data privacy requirements | — | A city plans to evaluate fairness in its citizen-facing chatbot. MLA-BiTe allows non-technical staff to create local scenario-based prompts to uncover discriminatory behaviour across sensitive categories, supporting multiple languages and augmentations. | | **Legal KG-RAG** | LIST | Proprietary | TRL 5–7 | LLM factuality accuracy testing | General Purpose AI | Transparency, Explainability, Robustness | Depends on whether the RAG is performed on sensitive data | — | A city using a standard RAG pipeline obtains irrelevant results. Legal KG-RAG rebuilds the legal corpus as a Neo4j knowledge graph, enabling direct comparison between traditional and KG-enhanced retrieval. | | **MLA-Reject** | LIST | To be open sourced | TRL 6–8 | LLM robustness to jailbreaking | General Purpose AI | Robustness | Depends on whether the system has access to sensitive data | — | A public administration operates a multilingual assistant for internal queries. They want to test robustness against unsafe or misleading prompts. MLA-Reject generates difficult negative prompts to test refusal behaviour and safety guardrails, revealing weaknesses and improving configurations. | From 7ae60420ad4daf08c6c4824b533a08687a8ba6cf Mon Sep 17 00:00:00 2001 From: alessiobuscemi Date: Thu, 27 Nov 2025 14:04:45 +0100 Subject: [PATCH 11/16] Update ai_assessment_catalogue.md --- docs/citcom_label/ai_assessment_catalogue.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/citcom_label/ai_assessment_catalogue.md b/docs/citcom_label/ai_assessment_catalogue.md index 031f5389..71d178de 100644 --- a/docs/citcom_label/ai_assessment_catalogue.md +++ b/docs/citcom_label/ai_assessment_catalogue.md @@ -47,19 +47,19 @@ If you would like to request an assessment or learn more about a tool, please co /* Make ALL columns narrow by default */ .md-typeset table:not(.no-format) th, .md-typeset table:not(.no-format) td { - width: 80px; + width: 60px; } -/* Make the Example of Use Case column (9th) wide */ +/* Make the Resources column (9th) small */ .md-typeset table:not(.no-format) th:nth-child(9), .md-typeset table:not(.no-format) td:nth-child(9) { - width: 450px; /* adjust as you want */ + width: 100px; } -/* Force the last column (10th: Resources) to be small */ +/* Make the last column (10th: Example of Use Case) wide */ .md-typeset table:not(.no-format) th:nth-child(10), .md-typeset table:not(.no-format) td:nth-child(10) { - width: 120px; /* prevents it from expanding */ + width: 500px; } From d4028048cc90e22c909d9614f9d60ba9fabc2e8d Mon Sep 17 00:00:00 2001 From: alessiobuscemi Date: Thu, 27 Nov 2025 14:05:59 +0100 Subject: [PATCH 12/16] Update ai_assessment_catalogue.md --- docs/citcom_label/ai_assessment_catalogue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/citcom_label/ai_assessment_catalogue.md b/docs/citcom_label/ai_assessment_catalogue.md index 71d178de..27ec93fb 100644 --- a/docs/citcom_label/ai_assessment_catalogue.md +++ b/docs/citcom_label/ai_assessment_catalogue.md @@ -67,7 +67,7 @@ If you would like to request an assessment or learn more about a tool, please co | Solution Name | Provider | Licensing Type | Project Phase / TRL | Domain of Application | AI Risk Category | Ethical Dimensions | Security & Securitization of Data | Resources | Example of Use Case | |---------------|----------|----------------|----------------------|------------------------|------------------|--------------------|-----------------------------------|-----------|----------------------| -| **FAIRGAME** | LIST | Open-source | TRL 6–8 | LLM bias testing, AI agents behavioural testing, jailbreaking testing | General Purpose AI | Fairness, Robustness | Depends on the use case (whether the chatbot/AI agent has access to sensitive data) | GitHub: , Paper: | A city aims to test its citizen-facing chatbot before launch. FAIRGAME enables the creation of simulated users with diverse identities, personalities, and requests using LLMs, allowing evaluation in dynamic, real-world-like conversations. | +| **FAIRGAME** | LIST | Open-source | TRL 6–8 | LLM bias testing, AI agents behavioural testing, jailbreaking testing | General Purpose AI | Fairness, Robustness | Depends on the use case (whether the chatbot/AI agent has access to sensitive data) | GitHub: , Paper: "FAIRGAME: A Framework for AI Agents Bias Recognition Using Game Theory", Frontiers in AI and Applications, Vol. 413: ECAI 2025| A city aims to test its citizen-facing chatbot before launch. FAIRGAME enables the creation of simulated users with diverse identities, personalities, and requests using LLMs, allowing evaluation in dynamic, real-world-like conversations. | | **MLA-BiTe** | LIST | To be open sourced | TRL 6–8 | LLM bias testing | General Purpose AI | Fairness, Robustness | No data privacy requirements | — | A city plans to evaluate fairness in its citizen-facing chatbot. MLA-BiTe allows non-technical staff to create local scenario-based prompts to uncover discriminatory behaviour across sensitive categories, supporting multiple languages and augmentations. | | **Legal KG-RAG** | LIST | Proprietary | TRL 5–7 | LLM factuality accuracy testing | General Purpose AI | Transparency, Explainability, Robustness | Depends on whether the RAG is performed on sensitive data | — | A city using a standard RAG pipeline obtains irrelevant results. Legal KG-RAG rebuilds the legal corpus as a Neo4j knowledge graph, enabling direct comparison between traditional and KG-enhanced retrieval. | | **MLA-Reject** | LIST | To be open sourced | TRL 6–8 | LLM robustness to jailbreaking | General Purpose AI | Robustness | Depends on whether the system has access to sensitive data | — | A public administration operates a multilingual assistant for internal queries. They want to test robustness against unsafe or misleading prompts. MLA-Reject generates difficult negative prompts to test refusal behaviour and safety guardrails, revealing weaknesses and improving configurations. | From 340252bd165cee132689a8fe1a91428ecb7c2822 Mon Sep 17 00:00:00 2001 From: alessiobuscemi Date: Thu, 27 Nov 2025 14:12:37 +0100 Subject: [PATCH 13/16] Update index.md --- docs/data_catalog/index.md | 36 +++++++++++++++++++++--------------- 1 file changed, 21 insertions(+), 15 deletions(-) diff --git a/docs/data_catalog/index.md b/docs/data_catalog/index.md index bc2f2611..d6fd4bd3 100644 --- a/docs/data_catalog/index.md +++ b/docs/data_catalog/index.md @@ -1,12 +1,12 @@ --- -icon: material/robot-search-outline -title: AI Assessment Catalogue +icon: material/store-search-outline +title: Data Catalog hide: - toc --- +
-The AI Assessment Catalogue is a centralized hub showcasing the evaluation tools, testing frameworks, and assessment solutions available across the TEF network. -It is regularly updated as new methodologies and tools become available at each TEF site. -If you would like to request an assessment or learn more about a tool, please click **Contact** or open the linked resources. +The data catalog is a centralized hub to keep track of available datasets. It is regularly updated to include new data as it becomes available in any TEF node. If you want access to any dataset, please click "Contact" to reach the owners. -!!! question "[How to add new assessment solutions?](./instructions.md)" +!!! question "[How to add new datasets?](./instructions.md)"
-[:simple-github: Add New Assessment Solution](https://github.com/CitComAI-Hub/CitComAI-Hub.github.io/issues/new/choose){:target="_blank" .md-button .md-button--primary-light } +[:simple-github: Add New Datasets ](https://github.com/CitComAI-Hub/CitComAI-Hub.github.io/issues/new/choose){:target="_blank" .md-button .md-button--primary-light }
@@ -36,7 +35,7 @@ If you would like to request an assessment or learn more about a tool, please cl - + -| Solution Name | Provider | Licensing Type | Project Phase / TRL | Domain of Application | AI Risk Category | Ethical Dimensions | Security & Securitization of Data | Example of Use Case | Resources | -|---------------|----------|----------------|----------------------|------------------------|------------------|--------------------|-----------------------------------|----------------------|-----------| -| **FAIRGAME** | LIST | Open-source | TRL 6–8 | LLM bias testing, AI agents behavioural testing, jailbreaking testing | General Purpose AI | Fairness, Robustness | Depends on the use case (whether the chatbot/AI agent has access to sensitive data) | A city aims to test its citizen-facing chatbot before launch. FAIRGAME enables the creation of simulated users with diverse identities, personalities, and requests using LLMs, allowing evaluation in dynamic, real-world-like conversations. | GitHub: , Paper: | -| **MLA-BiTe** | LIST | To be open sourced | TRL 6–8 | LLM bias testing | General Purpose AI | Fairness, Robustness | No data privacy requirements | A city plans to evaluate fairness in its citizen-facing chatbot. MLA-BiTe allows non-technical staff to create local scenario-based prompts to uncover discriminatory behaviour across sensitive categories, supporting multiple languages and augmentations. | — | -| **Legal KG-RAG** | LIST | Proprietary | TRL 5–7 | LLM factuality accuracy testing | General Purpose AI | Transparency, Explainability, Robustness | Depends on whether the RAG is performed on sensitive data | A city using a standard RAG pipeline obtains irrelevant results. Legal KG-RAG rebuilds the legal corpus as a Neo4j knowledge graph, enabling direct comparison between traditional and KG-enhanced retrieval. | — | -| **MLA-Reject** | LIST | To be open sourced | TRL 6–8 | LLM robustness to jailbreaking | General Purpose AI | Robustness | Depends on whether the system has access to sensitive data | A public administration operates a multilingual assistant for internal queries. They want to test robustness against unsafe or misleading prompts. MLA-Reject generates difficult negative prompts to test refusal behaviour and safety guardrails, revealing weaknesses and improving configurations. | — | +| Dataset | Super Node | TEF Node | Site | Data Model | Sampling Time | Historical | Owner | Get Access | +| ------- | ---------- | -------- | ---- | ---------- | ------------- | ---------- | ----- | ---------- | +| [Waste Container](./metadata_datasets/south_spain_valencia.md) | South | Spain | Valencia | [gitlab_vlci](https://gitlab.com/vlci-public/models-dades/wastemanagement/-/blob/main/WasteContainer/spec.md?ref_type=heads) | RealTime | From 2000 | València City Council | [Contact](https://valencia.opendatasoft.com/pages/home/) | +| [Weather Forecast](./metadata_datasets/south_spain_valencia.md) | South | Spain | Valencia | [gitlab_vlci](https://gitlab.com/vlci-public/models-dades/weather/blob/main/WeatherForecast/spec.md) | Daily | From 2010 | València City Council | [Contact](https://valencia.opendatasoft.com/pages/home/) | +| [Bikeparking stands in Aarhus City](./metadata_datasets/nordic_citcom_gtm.md) | Nordic | Denmark | GTM | *no specific value* | Ongoing | No | Aarhus Municipality | [City of Aarhus](https://www.opendata.dk/city-of-aarhus/cykelparkering_aarhus) | +| [Bike terminals in Aarhus, air and tools](./metadata_datasets/nordic_citcom_gtm.md) | Nordic | Denmark | GTM | *no specific value* | Ongoing | No | Aarhus Municipality | [Cykelterminal - Dataset](https://www.opendata.dk/city-of-aarhus/cykelterminal) | +| [Citybike locations in Aarhus](./metadata_datasets/nordic_citcom_gtm.md) | Nordic | Denmark | GTM | *no specific value* | *no specific value* | No | Aarhus Municipality | [Aarhus Bycykel - Dataset](https://www.opendata.dk/city-of-aarhus/aarhus-bycykel) | +| [Fast track bikeroutes in Aarhus](./metadata_datasets/nordic_citcom_gtm.md) | Nordic | Denmark | GTM | *no specific value* | Ongoing | No | Aarhus Municipality | [Supercykelsti i Aarhus Kommune - Dataset](https://www.opendata.dk/city-of-aarhus/supercykelsti) | +| [recreative bikeroutes in Aarhus](./metadata_datasets/nordic_citcom_gtm.md) | Nordic | Denmark | GTM | *no specific value* | Ongoing | No | Aarhus Municipality | [Rekreative cykelruter - Dataset](https://www.opendata.dk/city-of-aarhus/rekreative-cykelruter) | +| [AirQuality](./metadata_datasets/south_italy_uptown.md) | South | Italy | UpTown | *no specific value* | 30s | Yes | Politecnico di Milano | [neslab.it](https://www.neslab.it) | +| [Biodiversity](./metadata_datasets/south_italy_uptown.md) | South | Italy | UpTown | *no specific value* | 15m | Yes | Politecnico di Milano | [neslab.it](https://www.neslab.it) | +| [Energy distribution](./metadata_datasets/south_italy_uptown.md) | South | Italy | UpTown | *no specific value* | Hourly | Yes | A2A | [neslab.it](https://www.neslab.it) | +| [Archaeological Site](./metadata_datasets/south_italy_mithraeum-of-circus-maximus.md) | South | Italy | Mithraeum of Circus Maximus | *no specific value* | Real-time | Yes | Politecnico di Milano | [neslab.it](https://www.neslab.it) | From 5e28fb5d183b806940677308f0c08425d4a57d66 Mon Sep 17 00:00:00 2001 From: alessiobuscemi Date: Thu, 27 Nov 2025 14:13:29 +0100 Subject: [PATCH 14/16] Update ai_assessment_catalogue.md --- docs/citcom_label/ai_assessment_catalogue.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/citcom_label/ai_assessment_catalogue.md b/docs/citcom_label/ai_assessment_catalogue.md index 27ec93fb..20d2373e 100644 --- a/docs/citcom_label/ai_assessment_catalogue.md +++ b/docs/citcom_label/ai_assessment_catalogue.md @@ -20,6 +20,7 @@ hide:
+ The AI Assessment Catalogue showcases the evaluation tools, testing frameworks, and assessment solutions available across the Citcom.ai TEF network. It is regularly updated as new methodologies and tools become available at each TEF site. If you would like to request an assessment or learn more about a tool, please contact the relevant TEF sites. From 6c99bba9f2b06560dc9b914bec9c3ec726cbea3c Mon Sep 17 00:00:00 2001 From: alessio0208 Date: Thu, 27 Nov 2025 14:26:06 +0100 Subject: [PATCH 15/16] removed citcom_label.md from toolbox --- docs/toolbox/citcom_label.md | 52 ------------------------------------ 1 file changed, 52 deletions(-) delete mode 100644 docs/toolbox/citcom_label.md diff --git a/docs/toolbox/citcom_label.md b/docs/toolbox/citcom_label.md deleted file mode 100644 index d6c32386..00000000 --- a/docs/toolbox/citcom_label.md +++ /dev/null @@ -1,52 +0,0 @@ -# Citcom Label - -The Citcom Label is an initiative currently under development within Citcom.ai. Its goal is to create a trusted, recognisable signal that helps AI providers demonstrate responsible practices and gives buyers—especially public-sector actors such as smart cities—a clearer basis for evaluating and procuring AI solutions. - - -## What will the Citcom Label be? - -The label is envisioned as a **system of digital badges**, each representing a specific dimension of trustworthiness assessed during the evaluation process. -These badges would include a **watermark**, ensuring authenticity and preventing misuse. Each badge would be **verifiable through the Citcom Hub**, allowing external stakeholders to confirm its origin, evaluation status, and associated criteria. - -The Citcom badges are **not intended to function as legally binding conformity certificates under the AI Act**. Instead, they serve as **smart-city–oriented quality marks**, helping cities and other public authorities gain confidence in the AI solutions they consider adopting. - -For AI innovators, the Citcom badge system provides **independent third-party validation**, helping them promote their solutions and demonstrate that they meet recognised standards of trustworthiness. For cities and public buyers, the badges offer **clear, evidence-based guidance** to support more informed and transparent procurement decisions. - -## On what basis will the Citcom badges be awarded? - -The detailed criteria are still being developed with Citcom partners, but several guiding principles are emerging: - -### Completion of an evaluation -A badge is expected to be awarded only once a solution completes a structured assessment aligned with shared guidelines for the relevant dimension of trustworthiness. - -### Common methodology -Work is ongoing to define a coherent framework that determines how systems are qualified, how requirements translate into test cases, and how results are interpreted across different trust dimensions. - -### Success thresholds -Initial discussions point toward setting minimum quantitative and qualitative thresholds that vary by product type, maturity level, and the specific dimension being assessed. - -### Real-world validation -Evaluations are expected to rely on practical or pilot scenarios using the actual product, ensuring that results reflect real-world behaviour. - - -## Who will conduct the assessment and with which methodologies? - -The assessment behind each Citcom badge will be carried out by the participating TEF sites. Each site brings its own specialised methodologies, tools, and testing infrastructures, reflecting the diversity of technical expertise across the Citcom network. - -These assessment solutions cover different dimensions of trustworthiness and can be consulted through the **AI Assessment Catalogue**, available at the following link: - -**** - -The catalogue provides an overview of the available evaluation tools, test suites, and methodologies, enabling innovators to understand which capabilities are applied to their systems and helping cities see how specific trust dimensions are assessed. - -### Can an AI provider receive assessments across multiple TEF sites? - -Yes. If a solution would benefit from complementary expertise available across several TEF sites, an AI provider can undergo assessments in multiple locations. In such cases, the **first-contact TEF site** will coordinate the overall process. - -The coordinating TEF site will: -- connect with the additional TEF sites that carry out their assessments independently, -- ensure that each participating site manages its own contractual and operational responsibilities, -- consolidate the evaluation results into a unified report, -- and oversee the issuance of the Citcom badges corresponding to the dimensions assessed across all sites. - -This ensures a seamless experience for AI innovators while leveraging the full breadth of expertise across the TEF network. From 8b7afd723bdf6200804541414350c2b3c160c699 Mon Sep 17 00:00:00 2001 From: alessio0208 Date: Thu, 27 Nov 2025 15:55:00 +0100 Subject: [PATCH 16/16] renamed folder --- docs/{citcom_label => ai_assessment}/ai_assessment_catalogue.md | 0 docs/{citcom_label => ai_assessment}/index.md | 0 2 files changed, 0 insertions(+), 0 deletions(-) rename docs/{citcom_label => ai_assessment}/ai_assessment_catalogue.md (100%) rename docs/{citcom_label => ai_assessment}/index.md (100%) diff --git a/docs/citcom_label/ai_assessment_catalogue.md b/docs/ai_assessment/ai_assessment_catalogue.md similarity index 100% rename from docs/citcom_label/ai_assessment_catalogue.md rename to docs/ai_assessment/ai_assessment_catalogue.md diff --git a/docs/citcom_label/index.md b/docs/ai_assessment/index.md similarity index 100% rename from docs/citcom_label/index.md rename to docs/ai_assessment/index.md