Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions .cursor/rules/article-reference-sps.mdc
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
---
description: Marcação SPS de referências bibliográficas (ref-list / element-citation)
globs: reference/**/*
alwaysApply: true
---

# Lista de referências (SciELO SPS)

Ao marcar, gerar ou validar referências em XML/JSON neste projeto, seguir Critérios SciELO Brasil / SPS.

## Estrutura obrigatória

- `<ref-list>` em `<back>` (obrigatório em documentos indexáveis, exceto errata, retratação, adendo, preocupação, parecer).
- Cada `<ref>` contém **obrigatoriamente** `<mixed-citation>` e `<element-citation>`.
- `<element-citation>` exige `@publication-type`.

| Tag | Função |
|---|---|
| `mixed-citation` | Texto da referência como apresentado (espaços, pontuação). Só formatação: `bold`, `italic`, `sup`, `sub`. |
| `element-citation` | Marcação estruturada para métricas. Sem pontuação entre elementos filhos. |

## `@publication-type` permitidos

`book` · `confproc` · `data` · `database` · `journal` · `legal-doc` · `letter` · `newspaper` · `patent` · `preprint` · `report` · `software` · `thesis` · `webpage` · `other`

## Títulos por tipo

| Tipo | Tag de título |
|---|---|
| `journal` | `article-title` + `source` (periódico) |
| `book` (obra) | `source` |
| `book` (capítulo) | `part-title` (capítulo) + `source` (livro). **Não** usar `chapter-title`. |
| `data` | `data-title` + `source` |
| `confproc` | `conf-name` (+ `conf-loc`, `conf-date`, `conf-num`, `conf-sponsor` quando houver) |
| `webpage` | `source` |

## Regras de marcação

- Preferir `source` + `year` quando a norma bibliográfica for completa.
- Autoria: `person-group` com `@person-group-type` (`author`, `editor`, `translator`, `compiler`). Nome único → só `surname`. Instituição → `collab` (com `author`).
- `fpage` implica `lpage`.
- `size` → `@units="pages"`.
- `date-in-citation` → `@content-type="access-date"`.
- DOI: `pub-id` com `@pub-id-type="doi"`.
- URL: `ext-link` com `@ext-link-type="uri"` (ou `doi`). **No máximo um** `ext-link` em `element-citation` e em `mixed-citation`.
- Sem elemento específico → `comment`. Proibido `<comment><ext-link>…</ext-link></comment>`; permitido texto + `ext-link` dentro de `comment`.
- Não envolver o texto inteiro de um elemento só com `italic`/`bold` em `element-citation`.

## Fixtures e testes do app `reference`

- Corpus de entrada: `reference/fixtures/references.txt` (uma ref por linha) → constante `REFERENCES`.
- Golden JATS: `reference/fixtures/references.xml` (`ref-list`).
- Contagens de `.txt` e `.xml` devem coincidir (`test_eval_corpus_aligned`).

## Código local

- JSON → XML: `reference/data_utils.get_xml` (campo `reftype` = `@publication-type`).
- Ao evoluir o gerador, alinhar tags ao SPS (`part-title` para capítulos, tipos `data`/`software`/`legal-doc`, etc.).

## Referências externas

- Critérios SciELO Brasil 5.2.8.1 (XML / SPS)
- Guia de citação de dados de pesquisa
- Sample PubMed Central Citations
9 changes: 9 additions & 0 deletions .cursor/rules/model.mdc
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
description: Sobre modelo de IA
alwaysApply: true
---

# Não utilizar provedores de IA

- Não posso utilizar APIs de IA para marcação.
- Não é para utilizar API de ia nesse projeto somente modelo local
10 changes: 10 additions & 0 deletions .envs.example/.local/.django
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,13 @@ REDIS_URL=redis://redis:6379/0

# Celery
# ------------------------------------------------------------------------------

# Reference (Llama via HTTP — serviço ollama no local.yml)
# ------------------------------------------------------------------------------
REFERENCE_ENABLED=true
REFERENCE_URL=http://ollama:11434
REFERENCE_MODEL=llama3.2:3b
# REFERENCE_TOKEN=
# REFERENCE_TIMEOUT=300
# REFERENCE_BATCH_SIZE=10
# REFERENCE_NUM_CTX=8192
10 changes: 10 additions & 0 deletions .envs.example/.production/.django
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,13 @@ DJANGO_ALLOWED_HOSTS=*
# Sentry
# ------------------------------------------------------------------------------
SENTRY_DSN=

# Reference (Llama via HTTP — Ollama-compatible API)
# ------------------------------------------------------------------------------
REFERENCE_ENABLED=true
REFERENCE_URL=
REFERENCE_MODEL=llama3.2:3b
# REFERENCE_TOKEN=
# REFERENCE_TIMEOUT=300
# REFERENCE_BATCH_SIZE=10
# REFERENCE_NUM_CTX=8192
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ venv/
.envs
.envs.local
*.envs
.token

# Logs
*.log
Expand Down
42 changes: 37 additions & 5 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -63,14 +63,19 @@ django_createsuperuser: ## Create a superuser
django_bash: ## Open bash in django container
docker compose -f $(COMPOSE_FILE) run --rm django bash

test: ## Run tests (pytest default)
docker compose -f $(COMPOSE_FILE) run --rm django pytest --reuse-db
test: ## Run tests (pytest default, excludes llama eval)
docker compose -f $(COMPOSE_FILE) run --rm django pytest --reuse-db -m "not llama"

test-llama: ## Run Llama reference eval tests (requires Ollama)
docker compose -f $(COMPOSE_FILE) run --rm django pytest --reuse-db -m llama reference/tests/test_references.py

test-fast: ## Run tests (pytest failfast)
docker compose -f $(COMPOSE_FILE) run --rm django pytest -x --reuse-db
docker compose -f $(COMPOSE_FILE) run --rm django pytest -x --reuse-db -m "not llama"

test-cov: ## Run tests with coverage
docker compose -f $(COMPOSE_FILE) run --rm django pytest --reuse-db --cov=manuscripts --cov-report=term-missing manuscripts/tests
test-cov: ## Run tests with coverage (reference, fail under 100%)
docker compose -f $(COMPOSE_FILE) run --rm django pytest --reuse-db -m "not llama" \
--cov=reference --cov-report=term-missing --cov-fail-under=100 \
reference/tests

test-fresh: ## Recreate test database and run pytest
docker compose -f $(COMPOSE_FILE) run --rm django pytest --create-db
Expand Down Expand Up @@ -120,6 +125,33 @@ restore_data: ## Restore database from backup/latest.sql
volume_down: ## Remove all volumes
docker compose -f $(COMPOSE_FILE) down -v

############################################
## ollama / reference
############################################

REFERENCE_MODEL ?= llama3.2:3b

ollama_pull: ## Pull Llama model into local ollama container
docker compose -f $(COMPOSE_FILE) exec ollama ollama pull $(REFERENCE_MODEL)

############################################
## JWT
############################################

JWT_USERNAME ?=
JWT_PASSWORD ?=

bearer_token: ## eval "$$(make bearer_token JWT_USERNAME=u JWT_PASSWORD=p)" then curl -H "Authorization: Bearer $$TOKEN"
@test -n "$(JWT_USERNAME)" && test -n "$(JWT_PASSWORD)" || (echo 'Usage: eval "$$(make bearer_token JWT_USERNAME=user JWT_PASSWORD=pass)"' >&2 && exit 1)
@TOKEN=$$(docker compose -f $(COMPOSE_FILE) run --rm -T \
-e JWT_USERNAME=$(JWT_USERNAME) \
-e JWT_PASSWORD=$(JWT_PASSWORD) \
django python manage.py shell -c "from django.contrib.auth import authenticate; from rest_framework_simplejwt.tokens import RefreshToken; import os, sys; user = authenticate(username=os.environ['JWT_USERNAME'], password=os.environ['JWT_PASSWORD']); sys.exit(1) if not user else print(RefreshToken.for_user(user).access_token)" \
2>/dev/null | tr -d '\r' | grep -Eo 'eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+' | tail -n 1); \
if [ -z "$$TOKEN" ]; then echo "Failed to obtain JWT (check JWT_USERNAME/JWT_PASSWORD)" >&2; exit 1; fi; \
printf '%s\n' "$$TOKEN" > .token; \
printf "export TOKEN='%s'\n" "$$TOKEN"

############################################
## Cleanup
############################################
Expand Down
13 changes: 13 additions & 0 deletions config/api_router.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
from django.conf import settings
from rest_framework.routers import DefaultRouter, SimpleRouter

from reference.api.v1.views import ReferenceViewSet

if settings.DEBUG:
router = DefaultRouter()
else:
router = SimpleRouter()

router.register("reference", ReferenceViewSet, basename="reference")

urlpatterns = router.urls
1 change: 1 addition & 0 deletions config/menu.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
WAGTAIL_MENU_GROUPS_ORDER = [
"celery_wagtail",
"reference",
]


Expand Down
64 changes: 52 additions & 12 deletions config/settings/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)

import os
from datetime import timedelta
from pathlib import Path

import environ
Expand Down Expand Up @@ -40,7 +41,7 @@
"core.home",
"wagtail.contrib.forms",
"wagtail.contrib.redirects",
'wagtail.contrib.settings',
"wagtail.contrib.settings",
"wagtail_modeladmin",
"wagtail.embeds",
"wagtail.sites",
Expand All @@ -63,20 +64,23 @@
"django.contrib.sessions",
"django.contrib.messages",
"django.contrib.staticfiles",
"django_celery_results"
"django_celery_results",
]

THIRD_PARTY_APPS = [
"compressor",
"wagtailautocomplete",
"django_celery_beat",
"rest_framework",
"wagtail_json_widget",
]

LOCAL_APPS = [
"users",
"core",
"core_settings",
"xml_manager",
"reference",
]

INSTALLED_APPS = DJANGO_APPS + THIRD_PARTY_APPS + LOCAL_APPS + WAGTAIL
Expand All @@ -85,7 +89,7 @@
"django.contrib.sessions.middleware.SessionMiddleware",
"django.middleware.common.CommonMiddleware",
"django.middleware.csrf.CsrfViewMiddleware",
'django.middleware.locale.LocaleMiddleware',
"django.middleware.locale.LocaleMiddleware",
"django.contrib.auth.middleware.AuthenticationMiddleware",
"django.contrib.messages.middleware.MessageMiddleware",
"django.middleware.clickjacking.XFrameOptionsMiddleware",
Expand All @@ -106,7 +110,7 @@
"django.template.context_processors.request",
"django.contrib.auth.context_processors.auth",
"django.contrib.messages.context_processors.messages",
'wagtail.contrib.settings.context_processors.settings',
"wagtail.contrib.settings.context_processors.settings",
],
},
},
Expand Down Expand Up @@ -145,13 +149,13 @@
LANGUAGE_CODE = "en"

LANGUAGES = [
('pt-br', 'Português (Brasil)'),
('es', 'Español'),
('en', 'English'),
("pt-br", "Português (Brasil)"),
("es", "Español"),
("en", "English"),
]

LOCALE_PATHS = [
os.path.join(BASE_DIR, 'locale'),
os.path.join(BASE_DIR, "locale"),
]

TIME_ZONE = "UTC"
Expand Down Expand Up @@ -239,10 +243,22 @@
# This can be omitted to allow all files, but note that this may present a security risk
# if untrusted users are allowed to upload files -
# see https://docs.wagtail.org/en/stable/advanced_topics/deploying.html#user-uploaded-files
WAGTAILDOCS_EXTENSIONS = ['csv', 'docx', 'json', 'key', 'odt', 'pdf', 'pptx', 'rtf', 'txt', 'xlsx', 'zip']
WAGTAILDOCS_EXTENSIONS = [
"csv",
"docx",
"json",
"key",
"odt",
"pdf",
"pptx",
"rtf",
"txt",
"xlsx",
"zip",
]

# https://docs.djangoproject.com/en/dev/ref/settings/#auth-user-model
AUTH_USER_MODEL = 'users.CustomUser'
AUTH_USER_MODEL = "users.CustomUser"

# Celery
# ------------------------------------------------------------------------------
Expand All @@ -269,13 +285,13 @@
CELERY_BEAT_SCHEDULER = "django_celery_beat.schedulers:DatabaseScheduler"
# http://docs.celeryproject.org/en/latest/userguide/configuration.html
DJANGO_CELERY_BEAT_TZ_AWARE = False
#CELERY PROMETHEUS DASHBOARD
# CELERY PROMETHEUS DASHBOARD
# https://docs.celeryq.dev/en/stable/userguide/configuration.html#worker-send-task-events
CELERY_WORKER_SEND_TASK_EVENTS = True
# https://docs.celeryq.dev/en/stable/userguide/configuration.html#std-setting-task_send_sent_event
CELERY_SEND_TASK_SENT_EVENT = True
CELERYD_SEND_EVENTS = True
CE_BUCKETS=1,2.5,5,10,30,60,300,600,900,1800
CE_BUCKETS = 1, 2.5, 5, 10, 30, 60, 300, 600, 900, 1800

# Celery Results
# ------------------------------------------------------------------------------
Expand All @@ -285,3 +301,27 @@
CELERY_RESULT_EXTENDED = True

DATA_UPLOAD_MAX_NUMBER_FIELDS = 10000
SILENCED_SYSTEM_CHECKS = ["treebeard.E001"]

REST_FRAMEWORK = {
"DEFAULT_PAGINATION_CLASS": "rest_framework.pagination.PageNumberPagination",
"PAGE_SIZE": env.int("DRF_PAGE_SIZE", default=10),
"DEFAULT_AUTHENTICATION_CLASSES": (
"rest_framework_simplejwt.authentication.JWTAuthentication",
"rest_framework.authentication.SessionAuthentication",
),
}

SIMPLE_JWT = {
"AUTH_HEADER_TYPES": ("Bearer",),
"ACCESS_TOKEN_LIFETIME": timedelta(minutes=60),
"REFRESH_TOKEN_LIFETIME": timedelta(days=1),
}

REFERENCE_ENABLED = env.bool("REFERENCE_ENABLED", default=True)
REFERENCE_URL = env("REFERENCE_URL", default="")
REFERENCE_MODEL = env("REFERENCE_MODEL", default="llama3.2:3b")
REFERENCE_TIMEOUT = env.int("REFERENCE_TIMEOUT", default=300)
REFERENCE_TOKEN = env("REFERENCE_TOKEN", default="")
REFERENCE_BATCH_SIZE = env.int("REFERENCE_BATCH_SIZE", default=10)
REFERENCE_NUM_CTX = env.int("REFERENCE_NUM_CTX", default=8192)
10 changes: 10 additions & 0 deletions config/urls.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,24 @@
from django.conf.urls.static import static
from django.contrib import admin
from django.urls import include, path
from rest_framework_simplejwt.views import TokenObtainPairView, TokenRefreshView
from wagtail import urls as wagtail_urls
from wagtail.admin import urls as wagtailadmin_urls
from wagtail.documents import urls as wagtaildocs_urls

from config import api_router

urlpatterns = [
path("django-admin/", admin.site.urls),
path("admin/", include(wagtailadmin_urls)),
path("documents/", include(wagtaildocs_urls)),
path("api/v1/auth/token/", TokenObtainPairView.as_view(), name="token_obtain_pair"),
path(
"api/v1/auth/token/refresh/",
TokenRefreshView.as_view(),
name="token_refresh",
),
path("api/v1/", include(api_router)),
path("i18n/", include("django.conf.urls.i18n")),
] + static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

Expand Down
7 changes: 0 additions & 7 deletions core/wagtail_hooks.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,6 @@ def ensure_image_title(sender, instance, **kwargs):
pre_save.connect(ensure_image_title, sender=get_image_model())


@hooks.register("construct_main_menu")
def keep_only_sps_validation_menu(request, menu_items):
menu_items[:] = [
item for item in menu_items if item.name == "sps_package_validation"
]


@hooks.register("construct_help_menu")
def replace_help_menu_items(request, help_menu_items):
help_menu_items[:] = [
Expand Down
Loading