feat(flyer-import): add detailed product signals and display names
- Added `signals` and `displayNameDetailed` fields to FlyerItem model in Prisma schema - Introduced `FlyerImportSignals` type with origin countries, labels, quality flags, variant, and packaging - Added `displayNameDetailed` field to FlyerImportItem DTO and Flutter model - Implemented utility functions for signal extraction and display name building - Updated flyer import service to persist and return signals/category data - Enhanced Flutter UI to display detailed product information including badges for signals - Added new test coverage for signals persistence and display name generation - Added new import-common module for shared import utilities - Created database migration for new fields - Added Kilo plan for feature development
This commit is contained in:
Nils-Johan Gynther
@@ -0,0 +1,198 @@
|
||||
# Plan: Harmonisera flyer-import och kvitto-import
|
||||
|
||||
## Mål
|
||||
Implementera en gemensam importmodell och matchningspipeline så att flyer-import och kvitto-import beter sig så likt som möjligt, med fokus på:
|
||||
- Automatisk strukturering av namn/brand/vikt samt bundle-detaljer
|
||||
- Automatiskt kategoriupplösning (`categoryHint -> categoryId`)
|
||||
- Matchning mot befintliga produkter via normaliserade namn + signaler
|
||||
- Ingen automatisk skapning av produkter
|
||||
- Förberedelse för framtida automation via strukturerade signaler (`signals` JSON)
|
||||
|
||||
## Icke-mål (denna implementation)
|
||||
- Ingen auto-create av produkter i produktkatalog
|
||||
- Ingen ändring av övergripande UI-flöde (manuell import/validering kvar)
|
||||
- Ingen full omskrivning av receipt-import; vi extraherar och återanvänder delar stegvis
|
||||
|
||||
## Nuvarande gap (från kodbasen)
|
||||
1. `FlyerItem.categoryId` sätts till `null` i parse-flödet trots `categoryHint`.
|
||||
2. Flyer-matchning använder enklare strategi än receipt-import (färre regler/signalvikter).
|
||||
3. Ingen strukturerad lagring av ursprung/etiketter (t.ex. Sverige, Eko) i flyer.
|
||||
4. Bundleinformation finns men exponeras inte tydligt som detaljnamn i payload.
|
||||
5. Receipt och flyer använder olika “kontrakt” för mellanrepresentation.
|
||||
|
||||
## Övergripande design
|
||||
Inför en gemensam intern domänmodell för importerade rader (backend), och låt både flyer- och kvittoflöde mappa till den innan kategori/matchning.
|
||||
|
||||
### Gemensam intern modell (ny)
|
||||
`ImportedItemCandidate` (internt, ej API-brytande initialt):
|
||||
- `rawName`, `normalizedName`, `brand`
|
||||
- `weight`, `bundleWeight`, `isBundle`, `bundleItems`
|
||||
- `price`, `priceUnit`, `comparisonPrice`, `comparisonUnit`
|
||||
- `categoryHint`, `categoryId`
|
||||
- `matchedProductId`, `matchedProductName`, `matchedVia`, `matchConfidence`, `matchReasons`
|
||||
- `signals` (JSON):
|
||||
- `originCountries: string[]`
|
||||
- `labels: string[]` (ekologisk, laktosfri, etc)
|
||||
- `qualityFlags: string[]` (normaliserade flaggor, ex `eco`)
|
||||
- `variant: string | null`
|
||||
- `packaging: string | null`
|
||||
- `displayNameDetailed` (beräknat fält, kan persistas eller beräknas vid response)
|
||||
|
||||
## Faser och implementation
|
||||
|
||||
## Fas 1: Datamodell och migration
|
||||
1. Uppdatera `backend/prisma/schema.prisma`:
|
||||
- Lägg till `signals Json?` på `FlyerItem`
|
||||
- Lägg till `displayNameDetailed String?` på `FlyerItem`
|
||||
2. Skapa Prisma-migration.
|
||||
3. Säkerställ bakåtkompatibilitet:
|
||||
- Nullabla fält
|
||||
- Ingen ändring av befintliga constraints/index som bryter drift
|
||||
4. (Valfritt i samma fas) indexera vanligt använda JSON-signaler senare först efter verifierad nytta.
|
||||
|
||||
### Acceptanskriterier fas 1
|
||||
- Migration appliceras lokalt utan dataförlust.
|
||||
- Befintliga endpoints fungerar med gamla rader (`signals = null`).
|
||||
|
||||
## Fas 2: Gemensamma normaliserings-/signalverktyg
|
||||
1. Skapa gemensam utility-modul i backend, exempel:
|
||||
- `backend/src/import-common/import-item.types.ts`
|
||||
- `backend/src/import-common/import-signals.util.ts`
|
||||
- `backend/src/import-common/import-display-name.util.ts`
|
||||
2. Implementera signal-extraktion från textfält (`rawName`, `brand`, `offerText`):
|
||||
- Ursprungsländer till `originCountries`
|
||||
- Etiketter/märkningar till `labels`/`qualityFlags`
|
||||
- Pack-format till `packaging`
|
||||
3. Normalisera utan att förlora information:
|
||||
- Ta bort signalord från primär matchsträng men spara i `signals`
|
||||
- Ex: `Fläskytterfilé (Sverige)` -> matchsträng `flaskytterfile`, `signals.originCountries=["Sverige"]`
|
||||
4. Implementera `displayNameDetailed`:
|
||||
- Bundle: inkludera `bundleItems` i visningsnamn
|
||||
- Ex: `Kaptenens Favoriter (Chumlax 3x100g + Alaska pollock 3x100g)`
|
||||
|
||||
### Acceptanskriterier fas 2
|
||||
- Signals extraheras deterministiskt för kända mönster (Sverige/Tyskland/Eko/Ekologiskt).
|
||||
- `displayNameDetailed` genereras för bundles.
|
||||
|
||||
## Fas 3: Kategoriupplösning i flyer (paritet med kvitto)
|
||||
1. Extrahera/återanvänd kategori-regelmotorn från receipt-import till gemensam tjänst:
|
||||
- Ex: `backend/src/import-common/category-resolver.service.ts`
|
||||
2. Använd den i flyer-import efter normalisering:
|
||||
- `categoryHint` + signaltext + regler -> `categoryId`
|
||||
3. Prioritet:
|
||||
- Produktmatchad kategori (om säkert matchad produkt har kategori) kan väga högst
|
||||
- Annars regelbaserad kategori
|
||||
- Annars behåll `categoryHint` utan `categoryId`
|
||||
4. Specifika regler för kött/fläskytterfilé verifieras.
|
||||
|
||||
### Acceptanskriterier fas 3
|
||||
- `Fläskytterfilé` får korrekt `categoryId` i flyer-session.
|
||||
- `categoryId` sätts automatiskt för en betydande andel rader med tydlig signal.
|
||||
|
||||
## Fas 4: Matchningsparitet flyer <-> kvitto
|
||||
1. Bryt ut matchning till gemensam matcher (eller harmonisera algoritm):
|
||||
- alias exact
|
||||
- canonical/normalized exact
|
||||
- token/fuzzy
|
||||
- bonus för brand/weight/signalträffar
|
||||
2. Matchning ska använda signalrensad namnsträng + metadata:
|
||||
- Länder och eco-etiketter ska inte sabotera namnmatch
|
||||
3. Standardisera reason codes mellan flöden (så långt möjligt utan brytande API):
|
||||
- `alias_exact`, `normalized_exact`, `token_overlap:*`, `no_match`
|
||||
4. Behåll strikt policy: ingen auto-create produkt.
|
||||
|
||||
### Acceptanskriterier fas 4
|
||||
- Färre `no_match` på samma flyer-input jämfört med baseline.
|
||||
- Matchningsorsaker blir mer förklarbara och konsekventa.
|
||||
|
||||
## Fas 5: API/DTO och persistens
|
||||
1. Uppdatera flyer DTO:
|
||||
- `backend/src/flyer-import/dto/flyer-import.response.ts`
|
||||
- Lägg till `signals` och `displayNameDetailed`.
|
||||
2. Uppdatera persistens i `flyer-import.service.ts`:
|
||||
- Spara `signals`, `displayNameDetailed`, `categoryId`.
|
||||
3. Säkerställ att `getSession`, `getLatestSession`, `updateSessionItem` returnerar nya fält.
|
||||
4. Behåll kompatibilitet mot klient:
|
||||
- Nya fält adderas utan att ta bort befintliga.
|
||||
|
||||
### Acceptanskriterier fas 5
|
||||
- Response innehåller tydlig bundle-info och signaler per rad.
|
||||
- Inga regressions i existerande frontend-parsing.
|
||||
|
||||
## Fas 6: Frontend (flyer import-tab)
|
||||
1. Uppdatera domänmodeller i Flutter:
|
||||
- `flutter/lib/features/import/domain/flyer_import_item.dart`
|
||||
- ev. session/result-objekt
|
||||
2. Visa `displayNameDetailed` där tillgängligt, annars fallback `rawName`.
|
||||
3. Visa `bundleItems` tydligt i list-/detaljrad.
|
||||
4. Visa badge/metadata för signaler (`Sverige`, `Ekologisk`) utan att skriva över produktnamn.
|
||||
5. Säkerställ att manuellt urval till inköpslista fortsätter fungera.
|
||||
|
||||
### Acceptanskriterier fas 6
|
||||
- Bundle-rader är tydligare i UI.
|
||||
- Ursprung/eko syns som metadata.
|
||||
|
||||
## Fas 7: Teststrategi
|
||||
|
||||
### Backend enhetstester
|
||||
- `flyer-normalizer.service.spec.ts`
|
||||
- extraktion av `signals` (origin/labels)
|
||||
- bundle-detaljnamn
|
||||
- Ny kategori-resolver-spec
|
||||
- `Fläskytterfilé` -> köttkategori
|
||||
- `flyer-import.service.spec.ts`
|
||||
- `categoryId` sätts vid tydlig signal
|
||||
- `signals` och `displayNameDetailed` persisteras/returneras
|
||||
- Matchningstester
|
||||
- namn med land/eko matchar korrekt produkt
|
||||
|
||||
### Integrationstester
|
||||
- End-to-end parseAndMatch med representativ flyer-fixture.
|
||||
- Verifiera att inga produkter auto-skaps.
|
||||
- Verifiera att shopping-list insertion fungerar med/utan `matchedProductId`.
|
||||
|
||||
### Frontendtester
|
||||
- Serialisering av nya fält i import-session.
|
||||
- Rendering av `displayNameDetailed` + `bundleItems`.
|
||||
|
||||
## Fas 8: Mätning och rollout
|
||||
1. Lägg till enkel före/efter-mätning i logg/trace:
|
||||
- andel `no_match`
|
||||
- andel med satt `categoryId`
|
||||
2. Soft rollout via feature flag (om möjligt), annars stegvis release.
|
||||
3. Utvärdera verkliga flyer-sessioner innan vidare automatisering.
|
||||
|
||||
## Konkreta filer att ändra (planerad)
|
||||
- `backend/prisma/schema.prisma`
|
||||
- `backend/src/flyer-import/flyer-import.service.ts`
|
||||
- `backend/src/flyer-import/services/flyer-normalizer.service.ts`
|
||||
- `backend/src/flyer-import/dto/flyer-import.response.ts`
|
||||
- `backend/src/receipt-import/receipt-import.service.ts` (endast för extraktion/återanvändning av gemensamma delar)
|
||||
- Nya gemensamma filer under `backend/src/import-common/*`
|
||||
- `flutter/lib/features/import/domain/flyer_import_item.dart`
|
||||
- `flutter/lib/features/import/data/flyer_import_session.dart`
|
||||
- `flutter/lib/features/import/presentation/flyer_import_tab.dart`
|
||||
- Relevanta spec/test-filer i backend + flutter
|
||||
|
||||
## Risker och mitigering
|
||||
- Risk: API-kontraktsändringar bryter klient.
|
||||
- Mitigering: endast additive fält, fallback på gamla fält.
|
||||
- Risk: Felkategori vid aggressiva regler.
|
||||
- Mitigering: regelprioritet + reason-codes + tester för edge cases.
|
||||
- Risk: Övermatchning av produkter.
|
||||
- Mitigering: tröskelvärden + konservativ confidence för fuzzy.
|
||||
|
||||
## Leveransordning (rekommenderad)
|
||||
1. Fas 1–2 (schema + signals + utilities)
|
||||
2. Fas 3 (kategoriupplösning flyer)
|
||||
3. Fas 4 (matchningsparitet)
|
||||
4. Fas 5 (DTO/persistens)
|
||||
5. Fas 6 (frontend)
|
||||
6. Fas 7–8 (tester + mätning/rollout)
|
||||
|
||||
## Definition of Done
|
||||
- Flyer och kvitto använder samma centrala regler för kategorisering/matchning där möjligt.
|
||||
- Flyer-rader innehåller `signals` och tydligare produktrepresentation (`displayNameDetailed`, bundle-innehåll).
|
||||
- `categoryId` sätts automatiskt i flyer när tillräcklig signal finns (inkl. fläskytterfilé-fall).
|
||||
- Ingen automatisk produktskapning sker.
|
||||
- Tester uppdaterade och gröna.
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
-- AlterTable
|
||||
ALTER TABLE `FlyerItem`
|
||||
ADD COLUMN `signals` JSON NULL,
|
||||
ADD COLUMN `displayNameDetailed` VARCHAR(191) NULL;
|
||||
@@ -325,6 +325,8 @@ model FlyerItem {
|
||||
bundleWeight String?
|
||||
isBundle Boolean @default(false)
|
||||
bundleItems Json?
|
||||
signals Json?
|
||||
displayNameDetailed String?
|
||||
offerText String?
|
||||
parseConfidence Float
|
||||
parseReasons Json?
|
||||
|
||||
@@ -1,5 +1,13 @@
|
||||
export type FlyerImportMatchVia = 'alias' | 'exact' | 'token' | 'none';
|
||||
|
||||
export type FlyerImportSignals = {
|
||||
originCountries: string[];
|
||||
labels: string[];
|
||||
qualityFlags: string[];
|
||||
variant: string | null;
|
||||
packaging: string | null;
|
||||
};
|
||||
|
||||
export type FlyerReasonDescriptor = {
|
||||
code: string;
|
||||
kind: 'parse' | 'match';
|
||||
@@ -24,6 +32,8 @@ export type FlyerImportItem = {
|
||||
bundleWeight: string | null;
|
||||
isBundle: boolean;
|
||||
bundleItems: string[];
|
||||
displayNameDetailed: string | null;
|
||||
signals: FlyerImportSignals | null;
|
||||
offerText: string | null;
|
||||
isOffer: boolean;
|
||||
offerLimitText: string | null;
|
||||
|
||||
@@ -5,15 +5,18 @@ import { FlyerImportService } from './flyer-import.service';
|
||||
import { TextExtractorService } from './services/text-extractor.service';
|
||||
import { AiFlyerParserService } from './services/ai-flyer-parser.service';
|
||||
import { FlyerNormalizerService } from './services/flyer-normalizer.service';
|
||||
import { CategoriesModule } from '../categories/categories.module';
|
||||
import { CategoryResolverService } from '../import-common/category-resolver.service';
|
||||
|
||||
@Module({
|
||||
imports: [PrismaModule],
|
||||
imports: [PrismaModule, CategoriesModule],
|
||||
controllers: [FlyerImportController],
|
||||
providers: [
|
||||
FlyerImportService,
|
||||
TextExtractorService,
|
||||
AiFlyerParserService,
|
||||
FlyerNormalizerService,
|
||||
CategoryResolverService,
|
||||
],
|
||||
})
|
||||
export class FlyerImportModule {}
|
||||
|
||||
@@ -3,25 +3,44 @@ import { FlyerImportService } from './flyer-import.service';
|
||||
|
||||
describe('FlyerImportService', () => {
|
||||
const prismaMock = {
|
||||
product: {
|
||||
findMany: jest.fn(),
|
||||
},
|
||||
receiptAlias: {
|
||||
findMany: jest.fn(),
|
||||
},
|
||||
flyerSession: {
|
||||
findFirst: jest.fn(),
|
||||
findUnique: jest.fn(),
|
||||
create: jest.fn(),
|
||||
},
|
||||
flyerItem: {
|
||||
findUnique: jest.fn(),
|
||||
update: jest.fn(),
|
||||
create: jest.fn(),
|
||||
},
|
||||
aiTrace: {
|
||||
create: jest.fn(),
|
||||
},
|
||||
category: {
|
||||
findUnique: jest.fn(),
|
||||
},
|
||||
};
|
||||
|
||||
const createService = () =>
|
||||
const createService = (overrides?: {
|
||||
categoriesService?: any;
|
||||
categoryResolver?: any;
|
||||
textExtractor?: any;
|
||||
aiParser?: any;
|
||||
normalizer?: any;
|
||||
}) =>
|
||||
new FlyerImportService(
|
||||
prismaMock as any,
|
||||
{} as any,
|
||||
{} as any,
|
||||
{} as any,
|
||||
overrides?.categoriesService ?? { findFlattened: jest.fn().mockResolvedValue([]) },
|
||||
overrides?.categoryResolver ?? { resolveForFlyer: jest.fn().mockReturnValue(null) },
|
||||
overrides?.textExtractor ?? {},
|
||||
overrides?.aiParser ?? {},
|
||||
overrides?.normalizer ?? {},
|
||||
);
|
||||
|
||||
beforeEach(() => {
|
||||
@@ -78,6 +97,8 @@ describe('FlyerImportService', () => {
|
||||
bundleWeight: null,
|
||||
isBundle: false,
|
||||
bundleItems: [],
|
||||
displayNameDetailed: 'Tomat',
|
||||
signals: { originCountries: ['Sverige'], labels: [], qualityFlags: [], variant: null, packaging: null },
|
||||
offerText: 'Max 2 kop/hushall',
|
||||
parseConfidence: 0.9,
|
||||
parseReasons: ['ai_parsed'],
|
||||
@@ -97,6 +118,8 @@ describe('FlyerImportService', () => {
|
||||
expect(result.items).toHaveLength(1);
|
||||
expect(result.items[0].flyerItemId).toBe(99);
|
||||
expect(result.items[0].matchedVia).toBe('exact');
|
||||
expect(result.items[0].displayNameDetailed).toBe('Tomat');
|
||||
expect(result.items[0].signals?.originCountries).toEqual(['Sverige']);
|
||||
expect(result.items[0].parseReasonsDetailed[0].title).toBe('AI-tolkad rad');
|
||||
expect(result.items[0].matchReasonsDetailed[0].title).toBe('Exakt normaliserad matchning');
|
||||
expect(result.sourceAvailable).toBe(false);
|
||||
@@ -140,6 +163,162 @@ describe('FlyerImportService', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseAndMatch', () => {
|
||||
it('persists and returns signals/displayNameDetailed/categoryId in parse pipeline', async () => {
|
||||
prismaMock.product.findMany.mockResolvedValue([
|
||||
{ id: 11, name: 'Fläskytterfilé', canonicalName: 'Fläskytterfilé', categoryId: 7 },
|
||||
]);
|
||||
prismaMock.receiptAlias.findMany.mockResolvedValue([]);
|
||||
prismaMock.flyerSession.create.mockResolvedValue({ id: 200 });
|
||||
prismaMock.flyerItem.create
|
||||
.mockResolvedValueOnce({ id: 1001 });
|
||||
prismaMock.aiTrace.create.mockResolvedValue({ id: 1 });
|
||||
|
||||
const categoriesService = { findFlattened: jest.fn().mockResolvedValue([]) };
|
||||
const categoryResolver = { resolveForFlyer: jest.fn().mockReturnValue(7) };
|
||||
const textExtractor = { extractText: jest.fn().mockResolvedValue('raw flyer text') };
|
||||
const aiParser = {
|
||||
parseWithAI: jest.fn().mockResolvedValue({
|
||||
items: [
|
||||
{
|
||||
rawName: 'Fläskytterfilé (Sverige) EKO',
|
||||
normalizedName: 'flaskytterfile sverige eko',
|
||||
brand: 'Garant',
|
||||
category: 'Kött',
|
||||
price: 99.9,
|
||||
unit: 'kg',
|
||||
comparisonPrice: null,
|
||||
comparisonUnit: null,
|
||||
weight: '900g',
|
||||
bundleWeight: null,
|
||||
isBundle: true,
|
||||
bundleItems: ['Del 1', 'Del 2'],
|
||||
offer: 'ekologiskt från Sverige',
|
||||
confidence: 0.93,
|
||||
reasonCodes: ['ai_parsed'],
|
||||
},
|
||||
],
|
||||
trace: { prompt: null, rawOutput: null, chunkCount: 1, retryCount: 0 },
|
||||
}),
|
||||
};
|
||||
const normalizer = {
|
||||
normalize: jest.fn().mockReturnValue([
|
||||
{
|
||||
rawName: 'Fläskytterfilé (Sverige) EKO',
|
||||
normalizedName: 'flaskytterfile sverige eko',
|
||||
brand: 'Garant',
|
||||
categoryHint: 'Kött',
|
||||
price: 99.9,
|
||||
priceUnit: 'kg',
|
||||
comparisonPrice: null,
|
||||
comparisonUnit: null,
|
||||
weight: '900g',
|
||||
bundleWeight: null,
|
||||
isBundle: true,
|
||||
bundleItems: ['Del 1', 'Del 2'],
|
||||
offerText: 'ekologiskt från Sverige',
|
||||
parseConfidence: 0.93,
|
||||
parseReasons: ['ai_parsed'],
|
||||
},
|
||||
]),
|
||||
};
|
||||
|
||||
const service = createService({
|
||||
categoriesService,
|
||||
categoryResolver,
|
||||
textExtractor,
|
||||
aiParser,
|
||||
normalizer,
|
||||
});
|
||||
|
||||
const result = await service.parseAndMatch(
|
||||
{
|
||||
originalname: 'flyer.pdf',
|
||||
mimetype: 'application/pdf',
|
||||
size: 10,
|
||||
buffer: Buffer.from('pdf'),
|
||||
} as any,
|
||||
1,
|
||||
);
|
||||
|
||||
expect(result.items).toHaveLength(1);
|
||||
expect(result.items[0].displayNameDetailed).toBe('Fläskytterfilé (Sverige) EKO (Del 1 + Del 2)');
|
||||
expect(result.items[0].signals?.originCountries).toEqual(['Sverige']);
|
||||
expect(result.items[0].signals?.qualityFlags).toContain('eco');
|
||||
expect(result.items[0].categoryId).toBe(7);
|
||||
expect(result.items[0].normalizedName).toBe('flaskytterfile');
|
||||
expect(prismaMock.flyerItem.create).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
data: expect.objectContaining({
|
||||
displayNameDetailed: 'Fläskytterfilé (Sverige) EKO (Del 1 + Del 2)',
|
||||
categoryId: 7,
|
||||
signals: expect.objectContaining({ originCountries: ['Sverige'] }),
|
||||
}),
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it('logs warning when categories fallback is used', async () => {
|
||||
prismaMock.product.findMany.mockResolvedValue([]);
|
||||
prismaMock.receiptAlias.findMany.mockResolvedValue([]);
|
||||
prismaMock.flyerSession.create.mockResolvedValue({ id: 201 });
|
||||
prismaMock.flyerItem.create.mockResolvedValue({ id: 1002 });
|
||||
prismaMock.aiTrace.create.mockResolvedValue({ id: 2 });
|
||||
|
||||
const categoriesService = { findFlattened: jest.fn().mockRejectedValue(new Error('db down')) };
|
||||
const textExtractor = { extractText: jest.fn().mockResolvedValue('raw') };
|
||||
const aiParser = {
|
||||
parseWithAI: jest.fn().mockResolvedValue({
|
||||
items: [{ rawName: 'Tomat' }],
|
||||
trace: { prompt: null, rawOutput: null, chunkCount: 1, retryCount: 0 },
|
||||
}),
|
||||
};
|
||||
const normalizer = {
|
||||
normalize: jest.fn().mockReturnValue([
|
||||
{
|
||||
rawName: 'Tomat',
|
||||
normalizedName: 'tomat',
|
||||
brand: null,
|
||||
categoryHint: null,
|
||||
price: null,
|
||||
priceUnit: null,
|
||||
comparisonPrice: null,
|
||||
comparisonUnit: null,
|
||||
weight: null,
|
||||
bundleWeight: null,
|
||||
isBundle: false,
|
||||
bundleItems: [],
|
||||
offerText: null,
|
||||
parseConfidence: 0.9,
|
||||
parseReasons: ['ai_parsed'],
|
||||
},
|
||||
]),
|
||||
};
|
||||
|
||||
const service = createService({
|
||||
categoriesService,
|
||||
textExtractor,
|
||||
aiParser,
|
||||
normalizer,
|
||||
});
|
||||
const warnSpy = jest.spyOn((service as any).logger, 'warn');
|
||||
|
||||
await service.parseAndMatch(
|
||||
{
|
||||
originalname: 'flyer.pdf',
|
||||
mimetype: 'application/pdf',
|
||||
size: 10,
|
||||
buffer: Buffer.from('pdf'),
|
||||
} as any,
|
||||
1,
|
||||
);
|
||||
|
||||
expect(warnSpy).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Could not load categories for flyer import'),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('getLatestSession', () => {
|
||||
it('returns empty response when no sessions exist', async () => {
|
||||
prismaMock.flyerSession.findFirst.mockResolvedValue(null);
|
||||
|
||||
@@ -9,6 +9,7 @@ import {
|
||||
import { Prisma } from '@prisma/client';
|
||||
import { PrismaService } from '../prisma/prisma.service';
|
||||
import { normalizeName } from '../common/utils/normalize-name';
|
||||
import { CategoriesService } from '../categories/categories.service';
|
||||
import {
|
||||
FlyerImportItem,
|
||||
FlyerImportMatchVia,
|
||||
@@ -18,6 +19,10 @@ import { TextExtractorService } from './services/text-extractor.service';
|
||||
import { AiFlyerParserService } from './services/ai-flyer-parser.service';
|
||||
import { FlyerNormalizerService } from './services/flyer-normalizer.service';
|
||||
import { describeMatchReason, describeParseReason } from './services/reason-codes';
|
||||
import { CategoryResolverService } from '../import-common/category-resolver.service';
|
||||
import { buildDisplayNameDetailed } from '../import-common/import-display-name.util';
|
||||
import { extractImportSignals } from '../import-common/import-signals.util';
|
||||
import { ImportedItemSignals } from '../import-common/import-item.types';
|
||||
|
||||
type FlyerParseItem = {
|
||||
rawName: string;
|
||||
@@ -62,6 +67,7 @@ type ProductLite = {
|
||||
id: number;
|
||||
name: string;
|
||||
canonicalName: string | null;
|
||||
categoryId: number | null;
|
||||
};
|
||||
|
||||
@Injectable()
|
||||
@@ -72,6 +78,8 @@ export class FlyerImportService {
|
||||
|
||||
constructor(
|
||||
private readonly prisma: PrismaService,
|
||||
private readonly categoriesService: CategoriesService,
|
||||
private readonly categoryResolver: CategoryResolverService,
|
||||
private readonly textExtractor: TextExtractorService,
|
||||
private readonly aiParser: AiFlyerParserService,
|
||||
private readonly normalizer: FlyerNormalizerService,
|
||||
@@ -81,10 +89,10 @@ export class FlyerImportService {
|
||||
const startedAt = Date.now();
|
||||
const parsed = await this.parseViaInternal(file);
|
||||
|
||||
const [products, aliases] = await Promise.all([
|
||||
const [products, aliases, categories] = await Promise.all([
|
||||
this.prisma.product.findMany({
|
||||
where: { ownerId: userId, isActive: true },
|
||||
select: { id: true, name: true, canonicalName: true },
|
||||
select: { id: true, name: true, canonicalName: true, categoryId: true },
|
||||
}),
|
||||
this.prisma.receiptAlias.findMany({
|
||||
where: {
|
||||
@@ -92,6 +100,12 @@ export class FlyerImportService {
|
||||
},
|
||||
select: { receiptName: true, productId: true },
|
||||
}),
|
||||
this.categoriesService.findFlattened().catch((error) => {
|
||||
this.logger.warn(
|
||||
`Could not load categories for flyer import, proceeding without rule categories: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
return [];
|
||||
}),
|
||||
]);
|
||||
|
||||
const aliasToProduct = new Map<string, number>();
|
||||
@@ -109,20 +123,39 @@ export class FlyerImportService {
|
||||
}
|
||||
|
||||
const items: FlyerImportItem[] = parsed.items.map((item) => {
|
||||
const match = this.matchItem(item, products, aliasToProduct, productById);
|
||||
const signalData = extractImportSignals({
|
||||
rawName: item.rawName,
|
||||
brand: item.brand,
|
||||
offerText: item.offerText,
|
||||
});
|
||||
|
||||
const match = this.matchItem(item, signalData.normalizedMatchName, signalData.signals, products, aliasToProduct, productById);
|
||||
const signals = this.extractOfferSignals(item.offerText);
|
||||
const price = item.price ?? signals.price;
|
||||
const priceUnit = this.normalizeUnit(item.priceUnit) ?? signals.priceUnit;
|
||||
const comparisonPrice = item.comparisonPrice ?? signals.comparisonPrice;
|
||||
const comparisonUnit = this.normalizeUnit(item.comparisonUnit) ?? signals.comparisonUnit;
|
||||
const offerLimitText = this.extractOfferLimitText(item.offerText);
|
||||
const displayNameDetailed = buildDisplayNameDetailed({
|
||||
rawName: item.rawName,
|
||||
isBundle: item.isBundle,
|
||||
bundleItems: this.sanitizeBundleItems(item.bundleItems),
|
||||
});
|
||||
const categoryId = this.categoryResolver.resolveForFlyer({
|
||||
categories,
|
||||
signalText: [item.rawName, item.brand ?? '', item.offerText ?? ''].join(' ').trim(),
|
||||
categoryHint: item.category,
|
||||
matchedProductCategoryId: match.product?.categoryId ?? null,
|
||||
matchConfidence: match.confidence,
|
||||
});
|
||||
|
||||
return {
|
||||
flyerItemId: null,
|
||||
rawName: item.rawName,
|
||||
normalizedName: item.normalizedName,
|
||||
normalizedName: signalData.normalizedMatchName || item.normalizedName,
|
||||
brand: item.brand,
|
||||
category: item.category,
|
||||
categoryId: null,
|
||||
categoryId,
|
||||
price,
|
||||
priceUnit,
|
||||
comparisonPrice,
|
||||
@@ -131,6 +164,8 @@ export class FlyerImportService {
|
||||
bundleWeight: item.bundleWeight,
|
||||
isBundle: item.isBundle,
|
||||
bundleItems: this.sanitizeBundleItems(item.bundleItems),
|
||||
displayNameDetailed,
|
||||
signals: signalData.signals,
|
||||
offerText: item.offerText,
|
||||
isOffer: this.isOfferItem(item, signals.hasCampaignPattern),
|
||||
offerLimitText,
|
||||
@@ -146,6 +181,8 @@ export class FlyerImportService {
|
||||
};
|
||||
});
|
||||
|
||||
this.logImportMetrics(items);
|
||||
|
||||
const persistedItems = await this.persistSessionWithItems(userId, parsed.retailer, items, file);
|
||||
|
||||
await this.persistFlyerTrace({
|
||||
@@ -387,8 +424,7 @@ export class FlyerImportService {
|
||||
|
||||
const savedItems: FlyerImportItem[] = [];
|
||||
for (const item of items) {
|
||||
const created = await this.prisma.flyerItem.create({
|
||||
data: {
|
||||
const createData: Prisma.FlyerItemUncheckedCreateInput = {
|
||||
sessionId: session.id,
|
||||
rawName: item.rawName,
|
||||
normalizedName: item.normalizedName,
|
||||
@@ -404,6 +440,8 @@ export class FlyerImportService {
|
||||
bundleWeight: item.bundleWeight,
|
||||
isBundle: item.isBundle,
|
||||
bundleItems: item.bundleItems,
|
||||
displayNameDetailed: item.displayNameDetailed,
|
||||
signals: item.signals as Prisma.InputJsonValue,
|
||||
offerText: item.offerText,
|
||||
parseConfidence: item.parseConfidence,
|
||||
parseReasons: item.parseReasons,
|
||||
@@ -412,7 +450,10 @@ export class FlyerImportService {
|
||||
matchedVia: item.matchedVia,
|
||||
matchConfidence: item.matchConfidence,
|
||||
matchReasons: item.matchReasons,
|
||||
},
|
||||
};
|
||||
|
||||
const created = await this.prisma.flyerItem.create({
|
||||
data: createData,
|
||||
select: { id: true },
|
||||
});
|
||||
|
||||
@@ -433,6 +474,8 @@ export class FlyerImportService {
|
||||
|
||||
private matchItem(
|
||||
item: FlyerParseItem,
|
||||
normalizedMatchName: string,
|
||||
itemSignals: ImportedItemSignals,
|
||||
products: ProductLite[],
|
||||
aliasToProduct: Map<string, number>,
|
||||
productById: Map<number, ProductLite>,
|
||||
@@ -442,7 +485,7 @@ export class FlyerImportService {
|
||||
confidence: number;
|
||||
reasons: string[];
|
||||
} {
|
||||
const normalized = normalizeName(item.rawName || item.normalizedName);
|
||||
const normalized = normalizedMatchName || normalizeName(item.normalizedName || item.rawName);
|
||||
if (!normalized) {
|
||||
return { product: null, via: 'none', confidence: 0, reasons: ['empty_name'] };
|
||||
}
|
||||
@@ -472,12 +515,25 @@ export class FlyerImportService {
|
||||
}
|
||||
|
||||
let best: { product: ProductLite; confidence: number; overlap: number } | null = null;
|
||||
const itemTokens = this.tokenize(item.rawName);
|
||||
const itemTokens = this.tokenize(normalized);
|
||||
for (const product of products) {
|
||||
const productTokens = this.tokenize(product.canonicalName ?? product.name);
|
||||
const overlap = this.tokenOverlap(itemTokens, productTokens);
|
||||
if (overlap <= 0) continue;
|
||||
const confidence = Math.min(0.92, 0.5 + overlap * 0.4);
|
||||
|
||||
let confidence = Math.min(0.93, 0.48 + overlap * 0.42);
|
||||
if (this.hasBrandSignal(item.brand, product)) {
|
||||
confidence += 0.04;
|
||||
}
|
||||
if (this.hasWeightSignal(item.weight, product)) {
|
||||
confidence += 0.03;
|
||||
}
|
||||
if (this.hasQualitySignal(itemSignals, product)) {
|
||||
confidence += 0.03;
|
||||
}
|
||||
|
||||
confidence = Math.min(0.95, confidence);
|
||||
|
||||
if (!best || confidence > best.confidence) {
|
||||
best = { product, confidence, overlap };
|
||||
}
|
||||
@@ -521,6 +577,30 @@ export class FlyerImportService {
|
||||
return intersection / union;
|
||||
}
|
||||
|
||||
private hasBrandSignal(brand: string | null, product: ProductLite): boolean {
|
||||
if (!brand) return false;
|
||||
const normalizedBrand = normalizeName(brand);
|
||||
if (!normalizedBrand) return false;
|
||||
|
||||
const normalizedProduct = normalizeName(`${product.name} ${product.canonicalName ?? ''}`);
|
||||
return normalizedProduct.includes(normalizedBrand);
|
||||
}
|
||||
|
||||
private hasWeightSignal(weight: string | null, product: ProductLite): boolean {
|
||||
if (!weight) return false;
|
||||
const normalizedWeight = normalizeName(weight);
|
||||
if (!normalizedWeight) return false;
|
||||
|
||||
const normalizedProduct = normalizeName(`${product.name} ${product.canonicalName ?? ''}`);
|
||||
return normalizedProduct.includes(normalizedWeight);
|
||||
}
|
||||
|
||||
private hasQualitySignal(signals: ImportedItemSignals, product: ProductLite): boolean {
|
||||
if (!signals.qualityFlags.includes('eco')) return false;
|
||||
const normalizedProduct = normalizeName(`${product.name} ${product.canonicalName ?? ''}`);
|
||||
return /\beko\b|\bekolog/i.test(normalizedProduct);
|
||||
}
|
||||
|
||||
private isOfferItem(item: FlyerParseItem, hasCampaignPattern: boolean): boolean {
|
||||
return (
|
||||
item.price != null
|
||||
@@ -745,6 +825,8 @@ export class FlyerImportService {
|
||||
bundleWeight: string | null;
|
||||
isBundle: boolean;
|
||||
bundleItems: Prisma.JsonValue | null;
|
||||
displayNameDetailed?: string | null;
|
||||
signals?: Prisma.JsonValue | null;
|
||||
offerText: string | null;
|
||||
parseConfidence: number;
|
||||
parseReasons: Prisma.JsonValue | null;
|
||||
@@ -759,6 +841,24 @@ export class FlyerImportService {
|
||||
return value.map((entry) => String(entry));
|
||||
};
|
||||
|
||||
const toSignals = (value: Prisma.JsonValue | null | undefined): ImportedItemSignals | null => {
|
||||
if (!value || typeof value !== 'object' || Array.isArray(value)) return null;
|
||||
const record = value as Record<string, unknown>;
|
||||
const toArray = (key: string): string[] => {
|
||||
const maybeArray = record[key];
|
||||
if (!Array.isArray(maybeArray)) return [];
|
||||
return maybeArray.map((entry) => String(entry));
|
||||
};
|
||||
|
||||
return {
|
||||
originCountries: toArray('originCountries'),
|
||||
labels: toArray('labels'),
|
||||
qualityFlags: toArray('qualityFlags'),
|
||||
variant: typeof record.variant === 'string' ? record.variant : null,
|
||||
packaging: typeof record.packaging === 'string' ? record.packaging : null,
|
||||
};
|
||||
};
|
||||
|
||||
const normalizedMatchVia =
|
||||
item.matchedVia === 'alias' || item.matchedVia === 'exact' || item.matchedVia === 'token'
|
||||
? item.matchedVia
|
||||
@@ -784,6 +884,14 @@ export class FlyerImportService {
|
||||
bundleWeight: item.bundleWeight,
|
||||
isBundle: item.isBundle,
|
||||
bundleItems: this.sanitizeBundleItems(toStringArray(item.bundleItems)),
|
||||
displayNameDetailed:
|
||||
item.displayNameDetailed ??
|
||||
buildDisplayNameDetailed({
|
||||
rawName: item.rawName,
|
||||
isBundle: item.isBundle,
|
||||
bundleItems: this.sanitizeBundleItems(toStringArray(item.bundleItems)),
|
||||
}),
|
||||
signals: toSignals(item.signals),
|
||||
offerText: item.offerText,
|
||||
isOffer:
|
||||
item.price != null
|
||||
@@ -858,6 +966,8 @@ export class FlyerImportService {
|
||||
bundleWeight: string | null;
|
||||
isBundle: boolean;
|
||||
bundleItems: Prisma.JsonValue | null;
|
||||
displayNameDetailed?: string | null;
|
||||
signals?: Prisma.JsonValue | null;
|
||||
offerText: string | null;
|
||||
parseConfidence: number;
|
||||
parseReasons: Prisma.JsonValue | null;
|
||||
@@ -918,4 +1028,17 @@ export class FlyerImportService {
|
||||
.slice(0, this.MAX_BUNDLE_ITEMS)
|
||||
.map((entry) => entry.slice(0, this.MAX_BUNDLE_ITEM_LENGTH));
|
||||
}
|
||||
|
||||
private logImportMetrics(items: FlyerImportItem[]): void {
|
||||
if (items.length === 0) return;
|
||||
|
||||
const noMatchCount = items.filter((item) => item.matchReasons.includes('no_match')).length;
|
||||
const categoryAssignedCount = items.filter((item) => item.categoryId != null).length;
|
||||
const noMatchRatio = (noMatchCount / items.length) * 100;
|
||||
const categoryAssignedRatio = (categoryAssignedCount / items.length) * 100;
|
||||
|
||||
this.logger.log(
|
||||
`Flyer import metrics: no_match=${noMatchCount}/${items.length} (${noMatchRatio.toFixed(1)}%), category_id=${categoryAssignedCount}/${items.length} (${categoryAssignedRatio.toFixed(1)}%)`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
import { CategoryResolverService } from './category-resolver.service';
|
||||
|
||||
describe('CategoryResolverService', () => {
|
||||
const service = new CategoryResolverService();
|
||||
|
||||
const categories = [
|
||||
{ id: 1, name: 'Kött, chark & fågel', path: 'Kött, chark & fågel' },
|
||||
{ id: 2, name: 'Kött', path: 'Kött, chark & fågel > Kött' },
|
||||
{ id: 3, name: 'Fläsk', path: 'Kött, chark & fågel > Kött > Fläsk' },
|
||||
{ id: 4, name: 'Bröd', path: 'Bröd & kakor > Bröd' },
|
||||
];
|
||||
|
||||
it('resolves Fläskytterfilé to pork category', () => {
|
||||
const categoryId = service.resolveForFlyer({
|
||||
categories,
|
||||
signalText: 'Fläskytterfilé Sverige',
|
||||
categoryHint: null,
|
||||
matchedProductCategoryId: null,
|
||||
matchConfidence: 0,
|
||||
});
|
||||
|
||||
expect(categoryId).toBe(3);
|
||||
});
|
||||
|
||||
it('prefers matched product category when confidence is high', () => {
|
||||
const categoryId = service.resolveForFlyer({
|
||||
categories,
|
||||
signalText: 'Något annat',
|
||||
categoryHint: 'Bröd',
|
||||
matchedProductCategoryId: 99,
|
||||
matchConfidence: 0.95,
|
||||
});
|
||||
|
||||
expect(categoryId).toBe(99);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,114 @@
|
||||
import { Injectable } from '@nestjs/common';
|
||||
import { FlatCategory } from '../categories/categories.service';
|
||||
|
||||
type ResolveFlyerCategoryParams = {
|
||||
categories: FlatCategory[];
|
||||
signalText: string;
|
||||
categoryHint: string | null;
|
||||
matchedProductCategoryId: number | null;
|
||||
matchConfidence: number;
|
||||
};
|
||||
|
||||
@Injectable()
|
||||
export class CategoryResolverService {
|
||||
resolveForFlyer(params: ResolveFlyerCategoryParams): number | null {
|
||||
if (params.matchedProductCategoryId != null && params.matchConfidence >= 0.9) {
|
||||
return params.matchedProductCategoryId;
|
||||
}
|
||||
|
||||
const normalizedSignal = normalizeForRules(params.signalText);
|
||||
|
||||
if (hasPorkLikeSignal(normalizedSignal)) {
|
||||
const pork = this.resolvePorkCategory(params.categories);
|
||||
if (pork) return pork.id;
|
||||
}
|
||||
|
||||
if (hasBreadLikeSignal(normalizedSignal)) {
|
||||
const bread = this.resolveBreadCategory(params.categories);
|
||||
if (bread) return bread.id;
|
||||
}
|
||||
|
||||
if (!params.categoryHint) return null;
|
||||
return this.resolveByHint(params.categories, params.categoryHint)?.id ?? null;
|
||||
}
|
||||
|
||||
private resolveByHint(categories: FlatCategory[], categoryHint: string): FlatCategory | undefined {
|
||||
const normalizedHint = normalizeForRules(categoryHint);
|
||||
|
||||
return categories.find((category) => {
|
||||
const normalizedName = normalizeForRules(category.name);
|
||||
const normalizedPath = normalizeForRules(category.path);
|
||||
return normalizedName === normalizedHint || normalizedPath === normalizedHint;
|
||||
});
|
||||
}
|
||||
|
||||
private resolvePorkCategory(categories: FlatCategory[]): FlatCategory | undefined {
|
||||
return (
|
||||
categories.find(
|
||||
(category) =>
|
||||
category.name.toLowerCase() === 'fläsk' &&
|
||||
category.path.toLowerCase().startsWith('kött, chark & fågel > kött > '),
|
||||
) ||
|
||||
categories.find(
|
||||
(category) =>
|
||||
category.name.toLowerCase() === 'kött' &&
|
||||
category.path.toLowerCase() === 'kött, chark & fågel > kött',
|
||||
) ||
|
||||
categories.find((category) => category.path.toLowerCase() === 'kött, chark & fågel')
|
||||
);
|
||||
}
|
||||
|
||||
private resolveBreadCategory(categories: FlatCategory[]): FlatCategory | undefined {
|
||||
return (
|
||||
categories.find(
|
||||
(category) =>
|
||||
category.name.toLowerCase() === 'rostbröd' &&
|
||||
category.path.toLowerCase().startsWith('bröd & kakor > bröd > '),
|
||||
) ||
|
||||
categories.find(
|
||||
(category) =>
|
||||
category.name.toLowerCase() === 'bröd' &&
|
||||
category.path.toLowerCase() === 'bröd & kakor > bröd',
|
||||
) ||
|
||||
categories.find((category) => category.path.toLowerCase() === 'bröd & kakor')
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeForRules(value: string): string {
|
||||
return value
|
||||
.toLowerCase()
|
||||
.normalize('NFD')
|
||||
.replace(/[\u0300-\u036f]/g, '')
|
||||
.replace(/[^a-z0-9]+/g, ' ')
|
||||
.trim();
|
||||
}
|
||||
|
||||
function hasPorkLikeSignal(normalized: string): boolean {
|
||||
return (
|
||||
normalized.includes('bacon') ||
|
||||
normalized.includes('sidflask') ||
|
||||
normalized.includes('pancetta') ||
|
||||
normalized.includes('flask') ||
|
||||
normalized.includes('flaskytterfile') ||
|
||||
normalized.includes('ytterfile') ||
|
||||
normalized.includes('karre') ||
|
||||
normalized.includes('kotlett')
|
||||
);
|
||||
}
|
||||
|
||||
function hasBreadLikeSignal(normalized: string): boolean {
|
||||
return (
|
||||
/\brostbrod\b/.test(normalized) ||
|
||||
/\brost\s*n\s*toast\b/.test(normalized) ||
|
||||
/\broast\s*n\s*toast\b/.test(normalized) ||
|
||||
/\btoastbrod\b/.test(normalized) ||
|
||||
/\bformbrod\b/.test(normalized) ||
|
||||
/\blantbrod\b/.test(normalized) ||
|
||||
/\bfullkornsbrod\b/.test(normalized) ||
|
||||
/\bfranska\b/.test(normalized) ||
|
||||
/\blimpa\b/.test(normalized) ||
|
||||
/\bbrod\b/.test(normalized) ||
|
||||
/\btoast\b/.test(normalized)
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
export function buildDisplayNameDetailed(params: {
|
||||
rawName: string;
|
||||
isBundle: boolean;
|
||||
bundleItems: string[];
|
||||
}): string {
|
||||
const rawName = params.rawName.trim();
|
||||
if (!params.isBundle) return rawName;
|
||||
|
||||
const items = params.bundleItems
|
||||
.map((item) => item.trim())
|
||||
.filter((item) => item.length > 0);
|
||||
|
||||
if (items.length === 0) return rawName;
|
||||
return `${rawName} (${items.join(' + ')})`;
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
export type ImportedItemSignals = {
|
||||
originCountries: string[];
|
||||
labels: string[];
|
||||
qualityFlags: string[];
|
||||
variant: string | null;
|
||||
packaging: string | null;
|
||||
};
|
||||
|
||||
export type ImportedItemCandidate = {
|
||||
rawName: string;
|
||||
normalizedName: string;
|
||||
brand: string | null;
|
||||
weight: string | null;
|
||||
bundleWeight: string | null;
|
||||
isBundle: boolean;
|
||||
bundleItems: string[];
|
||||
price: number | null;
|
||||
priceUnit: string | null;
|
||||
comparisonPrice: number | null;
|
||||
comparisonUnit: string | null;
|
||||
categoryHint: string | null;
|
||||
categoryId: number | null;
|
||||
matchedProductId: number | null;
|
||||
matchedProductName: string | null;
|
||||
matchedVia: string;
|
||||
matchConfidence: number;
|
||||
matchReasons: string[];
|
||||
signals: ImportedItemSignals | null;
|
||||
displayNameDetailed: string | null;
|
||||
};
|
||||
|
||||
export const EMPTY_IMPORTED_SIGNALS: ImportedItemSignals = {
|
||||
originCountries: [],
|
||||
labels: [],
|
||||
qualityFlags: [],
|
||||
variant: null,
|
||||
packaging: null,
|
||||
};
|
||||
@@ -0,0 +1,45 @@
|
||||
import { buildDisplayNameDetailed } from './import-display-name.util';
|
||||
import { extractImportSignals } from './import-signals.util';
|
||||
|
||||
describe('import signals utilities', () => {
|
||||
it('extracts deterministic origin and eco labels', () => {
|
||||
const result = extractImportSignals({
|
||||
rawName: 'Fläskytterfilé (Sverige) EKO',
|
||||
brand: 'Garant',
|
||||
offerText: 'Ekologiskt kött från Sverige',
|
||||
});
|
||||
|
||||
expect(result.signals.originCountries).toEqual(['Sverige']);
|
||||
expect(result.signals.labels).toContain('Ekologisk');
|
||||
expect(result.signals.qualityFlags).toContain('eco');
|
||||
expect(result.normalizedMatchName).toBe('flaskytterfile');
|
||||
});
|
||||
|
||||
it('extracts Germany and keeps labels deterministic', () => {
|
||||
const result = extractImportSignals({
|
||||
rawName: 'Korv från Tyskland',
|
||||
offerText: 'Tysk kvalitet',
|
||||
});
|
||||
|
||||
expect(result.signals.originCountries).toEqual(['Tyskland']);
|
||||
expect(result.signals.labels).toEqual([]);
|
||||
});
|
||||
|
||||
it('builds detailed display name for bundle rows', () => {
|
||||
expect(
|
||||
buildDisplayNameDetailed({
|
||||
rawName: 'Kaptenens Favoriter',
|
||||
isBundle: true,
|
||||
bundleItems: ['Chumlax 3x100g', 'Alaska pollock 3x100g'],
|
||||
}),
|
||||
).toBe('Kaptenens Favoriter (Chumlax 3x100g + Alaska pollock 3x100g)');
|
||||
});
|
||||
|
||||
it('extracts storpack packaging signal', () => {
|
||||
const result = extractImportSignals({
|
||||
rawName: 'Kycklingfilé storpack',
|
||||
});
|
||||
|
||||
expect(result.signals.packaging).toBe('storpack');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,103 @@
|
||||
import { normalizeName } from '../common/utils/normalize-name';
|
||||
import {
|
||||
EMPTY_IMPORTED_SIGNALS,
|
||||
ImportedItemSignals,
|
||||
} from './import-item.types';
|
||||
|
||||
type SignalExtractionInput = {
|
||||
rawName: string;
|
||||
brand?: string | null;
|
||||
offerText?: string | null;
|
||||
};
|
||||
|
||||
const ORIGIN_COUNTRY_PATTERNS: Array<{ label: string; regex: RegExp }> = [
|
||||
{ label: 'Sverige', regex: /\b(sverige|svensk(t|a)?|sweden)\b/i },
|
||||
{ label: 'Tyskland', regex: /\b(tyskland|tysk(t|a)?|germany|deutschland)\b/i },
|
||||
{ label: 'Norge', regex: /\b(norge|norsk(t|a)?)\b/i },
|
||||
{ label: 'Danmark', regex: /\b(danmark|dansk(t|a)?)\b/i },
|
||||
{ label: 'Finland', regex: /\b(finland|finsk(t|a)?)\b/i },
|
||||
];
|
||||
|
||||
const LABEL_PATTERNS: Array<{ label: string; qualityFlag: string | null; regex: RegExp }> = [
|
||||
{ label: 'Ekologisk', qualityFlag: 'eco', regex: /\b(eko|ekologisk(t|a)?|organic)\b/i },
|
||||
{ label: 'Laktosfri', qualityFlag: 'lactose_free', regex: /\b(laktosfri(tt|a)?|lactose\s*free)\b/i },
|
||||
{ label: 'Glutenfri', qualityFlag: 'gluten_free', regex: /\b(glutenfri(tt|a)?|gluten\s*free)\b/i },
|
||||
{ label: 'Vegansk', qualityFlag: 'vegan', regex: /\b(vegansk(t|a)?|vegan)\b/i },
|
||||
{ label: 'Vegetarisk', qualityFlag: 'vegetarian', regex: /\b(vegetarisk(t|a)?|vegetarian)\b/i },
|
||||
];
|
||||
|
||||
export type SignalExtractionResult = {
|
||||
signals: ImportedItemSignals;
|
||||
normalizedMatchName: string;
|
||||
};
|
||||
|
||||
export function extractImportSignals(input: SignalExtractionInput): SignalExtractionResult {
|
||||
const text = [input.rawName, input.brand ?? '', input.offerText ?? '']
|
||||
.filter((part) => part.trim().length > 0)
|
||||
.join(' ');
|
||||
|
||||
const origins = ORIGIN_COUNTRY_PATTERNS
|
||||
.filter((pattern) => pattern.regex.test(text))
|
||||
.map((pattern) => pattern.label);
|
||||
|
||||
const labels = LABEL_PATTERNS
|
||||
.filter((pattern) => pattern.regex.test(text))
|
||||
.map((pattern) => pattern.label);
|
||||
|
||||
const qualityFlags = LABEL_PATTERNS
|
||||
.filter((pattern) => pattern.qualityFlag && pattern.regex.test(text))
|
||||
.map((pattern) => pattern.qualityFlag as string);
|
||||
|
||||
const packaging = resolvePackaging(text);
|
||||
const variant = extractVariant(input.rawName);
|
||||
|
||||
const signals: ImportedItemSignals = {
|
||||
...EMPTY_IMPORTED_SIGNALS,
|
||||
originCountries: Array.from(new Set(origins)),
|
||||
labels: Array.from(new Set(labels)),
|
||||
qualityFlags: Array.from(new Set(qualityFlags)),
|
||||
variant,
|
||||
packaging,
|
||||
};
|
||||
|
||||
const normalizedMatchName = normalizeForMatching(input.rawName);
|
||||
|
||||
return { signals, normalizedMatchName };
|
||||
}
|
||||
|
||||
export function normalizeForMatching(rawName: string): string {
|
||||
let cleaned = rawName;
|
||||
|
||||
for (const pattern of ORIGIN_COUNTRY_PATTERNS) {
|
||||
cleaned = cleaned.replace(pattern.regex, ' ');
|
||||
}
|
||||
|
||||
for (const pattern of LABEL_PATTERNS) {
|
||||
cleaned = cleaned.replace(pattern.regex, ' ');
|
||||
}
|
||||
|
||||
cleaned = cleaned.replace(/[()\[\]]/g, ' ');
|
||||
cleaned = cleaned.replace(/\s+/g, ' ').trim();
|
||||
return normalizeName(cleaned) || normalizeName(rawName);
|
||||
}
|
||||
|
||||
function resolvePackaging(text: string): string | null {
|
||||
const normalized = text.toLowerCase();
|
||||
if (/\b\d+\s*[x×]\s*\d+\s*(g|kg|ml|cl|dl|l)\b/.test(normalized)) {
|
||||
return 'multipack';
|
||||
}
|
||||
if (/\bstorpack\b/.test(normalized)) {
|
||||
return 'storpack';
|
||||
}
|
||||
if (/\b(2-pack|3-pack|4-pack|5-pack|6-pack|pack)\b/.test(normalized)) {
|
||||
return 'pack';
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function extractVariant(rawName: string): string | null {
|
||||
const variantMatch = rawName.match(/\(([^)]+)\)/);
|
||||
if (!variantMatch) return null;
|
||||
const value = variantMatch[1].trim();
|
||||
return value.length > 0 ? value : null;
|
||||
}
|
||||
@@ -1,13 +1,58 @@
|
||||
import 'flyer_reason_descriptor.dart';
|
||||
|
||||
class FlyerImportSignals {
|
||||
final List<String> originCountries;
|
||||
final List<String> labels;
|
||||
final List<String> qualityFlags;
|
||||
final String? variant;
|
||||
final String? packaging;
|
||||
|
||||
const FlyerImportSignals({
|
||||
this.originCountries = const [],
|
||||
this.labels = const [],
|
||||
this.qualityFlags = const [],
|
||||
this.variant,
|
||||
this.packaging,
|
||||
});
|
||||
|
||||
factory FlyerImportSignals.fromJson(Map<String, dynamic> json) {
|
||||
return FlyerImportSignals(
|
||||
originCountries:
|
||||
(json['originCountries'] as List?)?.map((e) => e.toString()).toList() ?? const [],
|
||||
labels: (json['labels'] as List?)?.map((e) => e.toString()).toList() ?? const [],
|
||||
qualityFlags:
|
||||
(json['qualityFlags'] as List?)?.map((e) => e.toString()).toList() ?? const [],
|
||||
variant: json['variant'] as String?,
|
||||
packaging: json['packaging'] as String?,
|
||||
);
|
||||
}
|
||||
|
||||
Map<String, dynamic> toJson() {
|
||||
return {
|
||||
'originCountries': originCountries,
|
||||
'labels': labels,
|
||||
'qualityFlags': qualityFlags,
|
||||
'variant': variant,
|
||||
'packaging': packaging,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
class FlyerImportItem {
|
||||
final int? flyerItemId;
|
||||
final String rawName;
|
||||
final String? displayNameDetailed;
|
||||
final String normalizedName;
|
||||
final String? brand;
|
||||
final String? category;
|
||||
final int? categoryId;
|
||||
final double? price;
|
||||
final String? priceUnit;
|
||||
final String? weight;
|
||||
final String? bundleWeight;
|
||||
final bool isBundle;
|
||||
final List<String> bundleItems;
|
||||
final FlyerImportSignals? signals;
|
||||
final String? offerText;
|
||||
final bool isOffer;
|
||||
final String? offerLimitText;
|
||||
@@ -26,11 +71,18 @@ class FlyerImportItem {
|
||||
FlyerImportItem({
|
||||
required this.flyerItemId,
|
||||
required this.rawName,
|
||||
this.displayNameDetailed,
|
||||
required this.normalizedName,
|
||||
this.brand,
|
||||
this.category,
|
||||
this.categoryId,
|
||||
this.price,
|
||||
this.priceUnit,
|
||||
this.weight,
|
||||
this.bundleWeight,
|
||||
this.isBundle = false,
|
||||
this.bundleItems = const [],
|
||||
this.signals,
|
||||
this.offerText,
|
||||
this.isOffer = false,
|
||||
this.offerLimitText,
|
||||
@@ -51,11 +103,20 @@ class FlyerImportItem {
|
||||
return FlyerImportItem(
|
||||
flyerItemId: (json['flyerItemId'] as num?)?.toInt(),
|
||||
rawName: json['rawName'] as String? ?? '',
|
||||
displayNameDetailed: json['displayNameDetailed'] as String?,
|
||||
normalizedName: json['normalizedName'] as String? ?? '',
|
||||
brand: json['brand'] as String?,
|
||||
category: json['category'] as String?,
|
||||
categoryId: (json['categoryId'] as num?)?.toInt(),
|
||||
price: (json['price'] as num?)?.toDouble(),
|
||||
priceUnit: json['priceUnit'] as String?,
|
||||
weight: json['weight'] as String?,
|
||||
bundleWeight: json['bundleWeight'] as String?,
|
||||
isBundle: json['isBundle'] == true,
|
||||
bundleItems: (json['bundleItems'] as List?)?.map((e) => e.toString()).toList() ?? const [],
|
||||
signals: json['signals'] is Map
|
||||
? FlyerImportSignals.fromJson(Map<String, dynamic>.from(json['signals'] as Map))
|
||||
: null,
|
||||
offerText: json['offerText'] as String?,
|
||||
isOffer: json['isOffer'] == true,
|
||||
offerLimitText: json['offerLimitText'] as String?,
|
||||
@@ -87,11 +148,18 @@ class FlyerImportItem {
|
||||
return {
|
||||
'flyerItemId': flyerItemId,
|
||||
'rawName': rawName,
|
||||
'displayNameDetailed': displayNameDetailed,
|
||||
'normalizedName': normalizedName,
|
||||
'brand': brand,
|
||||
'category': category,
|
||||
'categoryId': categoryId,
|
||||
'price': price,
|
||||
'priceUnit': priceUnit,
|
||||
'weight': weight,
|
||||
'bundleWeight': bundleWeight,
|
||||
'isBundle': isBundle,
|
||||
'bundleItems': bundleItems,
|
||||
'signals': signals?.toJson(),
|
||||
'offerText': offerText,
|
||||
'isOffer': isOffer,
|
||||
'offerLimitText': offerLimitText,
|
||||
@@ -119,11 +187,18 @@ class FlyerImportItem {
|
||||
return FlyerImportItem(
|
||||
flyerItemId: flyerItemId,
|
||||
rawName: rawName ?? this.rawName,
|
||||
displayNameDetailed: displayNameDetailed,
|
||||
normalizedName: normalizedName,
|
||||
brand: brand,
|
||||
category: category ?? this.category,
|
||||
categoryId: categoryId ?? this.categoryId,
|
||||
price: price,
|
||||
priceUnit: priceUnit,
|
||||
weight: weight,
|
||||
bundleWeight: bundleWeight,
|
||||
isBundle: isBundle,
|
||||
bundleItems: bundleItems,
|
||||
signals: signals,
|
||||
offerText: offerText,
|
||||
isOffer: isOffer,
|
||||
offerLimitText: offerLimitText,
|
||||
|
||||
@@ -501,6 +501,40 @@ class _FlyerImportTabState extends ConsumerState<FlyerImportTab> {
|
||||
);
|
||||
}
|
||||
|
||||
Widget _buildMetadataBadges(FlyerImportItem item, ThemeData theme) {
|
||||
final values = <String>[
|
||||
...(item.signals?.originCountries ?? const <String>[]),
|
||||
...(item.signals?.labels ?? const <String>[]),
|
||||
];
|
||||
if (values.isEmpty) return const SizedBox.shrink();
|
||||
|
||||
return Wrap(
|
||||
spacing: 6,
|
||||
runSpacing: 6,
|
||||
children: values
|
||||
.map(
|
||||
(value) => Container(
|
||||
padding: const EdgeInsets.symmetric(horizontal: 8, vertical: 3),
|
||||
decoration: BoxDecoration(
|
||||
color: theme.colorScheme.primary.withValues(alpha: 0.08),
|
||||
borderRadius: BorderRadius.circular(999),
|
||||
border: Border.all(
|
||||
color: theme.colorScheme.primary.withValues(alpha: 0.25),
|
||||
),
|
||||
),
|
||||
child: Text(
|
||||
value,
|
||||
style: theme.textTheme.labelSmall?.copyWith(
|
||||
color: theme.colorScheme.primary,
|
||||
fontWeight: FontWeight.w600,
|
||||
),
|
||||
),
|
||||
),
|
||||
)
|
||||
.toList(),
|
||||
);
|
||||
}
|
||||
|
||||
Future<void> _copyText(String value, String label) async {
|
||||
await Clipboard.setData(ClipboardData(text: value));
|
||||
if (!mounted) return;
|
||||
@@ -744,7 +778,7 @@ class _FlyerImportTabState extends ConsumerState<FlyerImportTab> {
|
||||
},
|
||||
title: Row(
|
||||
children: [
|
||||
Expanded(child: Text(item.rawName)),
|
||||
Expanded(child: Text(item.displayNameDetailed ?? item.rawName)),
|
||||
IconButton(
|
||||
tooltip: 'Redigera',
|
||||
visualDensity: VisualDensity.compact,
|
||||
@@ -765,6 +799,10 @@ class _FlyerImportTabState extends ConsumerState<FlyerImportTab> {
|
||||
crossAxisAlignment: CrossAxisAlignment.start,
|
||||
children: [
|
||||
if (priceText.isNotEmpty) Text('Pris: $priceText'),
|
||||
if (item.isBundle && item.bundleItems.isNotEmpty)
|
||||
Text('Paketinnehåll: ${item.bundleItems.join(' + ')}'),
|
||||
if (item.brand != null && item.brand!.trim().isNotEmpty)
|
||||
Text('Varumärke: ${item.brand}'),
|
||||
if ((item.category ?? '').trim().isNotEmpty)
|
||||
Text('Kategori: ${item.category}'),
|
||||
if (comparisonText.isNotEmpty)
|
||||
@@ -778,6 +816,11 @@ class _FlyerImportTabState extends ConsumerState<FlyerImportTab> {
|
||||
),
|
||||
),
|
||||
if (sanitizedOfferText.isNotEmpty) Text(sanitizedOfferText),
|
||||
if ((item.signals?.originCountries.isNotEmpty ?? false) ||
|
||||
(item.signals?.labels.isNotEmpty ?? false)) ...[
|
||||
const SizedBox(height: 6),
|
||||
_buildMetadataBadges(item, theme),
|
||||
],
|
||||
if (item.matchedProductName != null)
|
||||
Text('Match: ${item.matchedProductName}'),
|
||||
if (detailedReasons.isNotEmpty) ...[
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
import 'package:flutter_test/flutter_test.dart';
|
||||
import 'package:recipe_flutter/features/import/domain/flyer_import_item.dart';
|
||||
|
||||
void main() {
|
||||
test('parses signals and detailed bundle name', () {
|
||||
final item = FlyerImportItem.fromJson({
|
||||
'flyerItemId': 7,
|
||||
'rawName': 'Kaptenens Favoriter',
|
||||
'displayNameDetailed':
|
||||
'Kaptenens Favoriter (Chumlax 3x100g + Alaska pollock 3x100g)',
|
||||
'normalizedName': 'kaptenens favoriter',
|
||||
'isBundle': true,
|
||||
'bundleItems': ['Chumlax 3x100g', 'Alaska pollock 3x100g'],
|
||||
'signals': {
|
||||
'originCountries': ['Sverige'],
|
||||
'labels': ['Ekologisk'],
|
||||
'qualityFlags': ['eco'],
|
||||
'variant': null,
|
||||
'packaging': 'multipack',
|
||||
},
|
||||
});
|
||||
|
||||
expect(item.isBundle, isTrue);
|
||||
expect(item.displayNameDetailed, contains('Chumlax 3x100g'));
|
||||
expect(item.bundleItems, hasLength(2));
|
||||
expect(item.signals?.originCountries, ['Sverige']);
|
||||
expect(item.toJson()['signals'], isA<Map<String, dynamic>>());
|
||||
});
|
||||
}
|
||||
Reference in New Issue
Block a user