import CallToActionHashMapFake from "../fake/targeting/CallToActionHashMapFake"; import AudienceHashMapFake from "../fake/targeting/AudienceHashMapFake"; import { expect } from "chai"; import { Campaign, ChoiceRequest } from "../../../src/models"; import { MockedFeatureConfig } from "@ogury/feature-config-sdk"; import SpeedService from "../../../src/services/speed.service"; import { FeatureConfigService } from "../../../src/services/FeatureConfig.service"; import OrderingStep from "../../../src/steps/ordering/ordering.step"; import { FakeRequestBuilder } from "../fakeRequestBuilder"; import { CampaignCreative, CompositeScoreStrategy, Weights, } from "../../../src/steps/ordering/ordering.model"; import constants from "../../../src/utils/constants"; import { BucketMapping } from "../../../src/services/scoringSystem/abstractScoreHashMap.service"; import * as helper from "../../../src/utils/helper"; import { bisectRight } from "../../../src/utils/helper"; import { CampaignModelFake } from "../fake/CampaignModelFake"; import { CampaignData } from "../../../src/models/CampaignModel"; import { FakeCampaign } from "../fake/Campaign"; import sinon from "sinon"; const ContextServiceFake = require("../fake/targeting/ContextServiceFake"); /** * mocked campaigns: * has context score * 1: no * 2: no * 3: no * 4: no * 5: yes * 6: yes * 7: yes * 8: yes */ describe("Ordering step", function () { let campaignModel: CampaignModelFake; let contextService: any; let callToActionHashMap: CallToActionHashMapFake; let audienceHashMap: AudienceHashMapFake; let candidateCampaigns: Campaign[]; let campaignData: Partial[]; let featureConfigService: FeatureConfigService; let mockedFeatureConf: MockedFeatureConfig; let speedService: SpeedService; let orderingStep: OrderingStep; // todo: context: test hashmap part // todo: context: test ML part before(() => { /** * mocked campaigns: * has context score * 1: no * 2: no * 3: no * 4: no * 5: yes * 6: yes * 7: yes * 8: yes */ campaignModel = new CampaignModelFake(); contextService = new ContextServiceFake({ "1": { "11": { score: 0.1, bucketsMap: [{ inputUpper: 1, outputUpper: 1, outputLower: 1 }] }, }, "2": { "21": { score: 0.2, bucketsMap: [{ inputUpper: 2, outputUpper: 2, outputLower: 2 }] }, }, "3": { "31": { score: 0.3, bucketsMap: [{ inputUpper: 3, outputUpper: 3, outputLower: 3 }] }, }, "4": { "41": { score: 0.4, bucketsMap: [{ inputUpper: 4, outputUpper: 4, outputLower: 4 }] }, }, "5": { "51": { score: 0.5, bucketsMap: [{ inputUpper: 5, outputUpper: 5, outputLower: 5 }] }, }, "6": { "61": { score: 0.6, bucketsMap: [{ inputUpper: 6, outputUpper: 6, outputLower: 6 }] }, }, "7": { "71": { score: 0.7, bucketsMap: [{ inputUpper: 7, outputUpper: 7, outputLower: 7 }] }, }, "8": { "81": { score: 0.8, bucketsMap: [{ inputUpper: 8, outputUpper: 8, outputLower: 8 }] }, }, }); callToActionHashMap = CallToActionHashMapFake.fromCtaMaps({ scoresMap: new Map([ ["-1|ogury|5|an.app|aui1", 0.25], ["-1|ogury|6|-1|-1", 0.3], ["-1|adru|6|-1|-1", 0.3], ["-1|ogury|7|an.app|aui1", 0.35], ["-1|ogury|8|-1|-1", 0.4], ["-1|adru|8|-1|-1", 0.4], ["MPU|-1|-1|-1|-1", 0.5], ]), normMap: new Map([ [ "-1|1", { bucketsMap: [ { inputUpper: 0.1, outputLower: 0, outputUpper: 0.76, }, { inputUpper: 0.785, outputLower: 0.76, outputUpper: 0.91, }, { inputUpper: 1, outputLower: 0.91, outputUpper: 1, }, ], }, ], [ "-1|2", { bucketsMap: [ { inputUpper: 0.1, outputLower: 0, outputUpper: 0.76, }, { inputUpper: 0.785, outputLower: 0.76, outputUpper: 0.92, }, { inputUpper: 1, outputLower: 0.92, outputUpper: 1, }, ], }, ], [ "MPU|-1", { bucketsMap: [ { inputUpper: 0.1, outputLower: 0, outputUpper: 0.76, }, { inputUpper: 0.785, outputLower: 0.76, outputUpper: 0.93, }, { inputUpper: 1, outputLower: 0.93, outputUpper: 1, }, ], }, ], [ "-1|-1", { bucketsMap: [ { inputUpper: 0.1, outputLower: 0, outputUpper: 0.76, }, { inputUpper: 0.785, outputLower: 0.76, outputUpper: 0.94, }, { inputUpper: 1, outputLower: 0.94, outputUpper: 1, }, ], }, ], ]), }); audienceHashMap = AudienceHashMapFake.fromAudienceScores({ apps: new Map([ ["an.app", 0], ["appBundle", 1], ]), campaigns: new Map([ ["1", "0"], ["2", "0"], ["3", "0"], ["4", "0"], ["5", "1"], ["6", "1"], ["7", "1"], ["8", "1"], ]), similarities: new Map([ [ "0", new Map([ [ 0, { score: 0.1, bucketsMap: [{ inputUpper: 0.1, outputUpper: 0.1, outputLower: 0.1 }], }, ], [ 1, { score: 0.11, bucketsMap: [{ inputUpper: 0.11, outputUpper: 0.11, outputLower: 0.11 }], }, ], [ -1, { score: 0.105, bucketsMap: [{ inputUpper: 0.105, outputUpper: 0.105, outputLower: 0.105 }], }, ], ]), ], [ "1", new Map([ [ 0, { score: 0.2, bucketsMap: [{ inputUpper: 0.2, outputUpper: 0.2, outputLower: 0.2 }] }, ], [ 1, { score: 0.22, bucketsMap: [{ inputUpper: 0.22, outputUpper: 0.22, outputLower: 0.22 }], }, ], [ -1, { score: 0.21, bucketsMap: [{ inputUpper: 0.21, outputUpper: 0.21, outputLower: 0.21 }], }, ], ]), ], [ "-1", new Map([ [ -1, { score: 0.1575, bucketsMap: [{ inputUpper: 0.1575, outputUpper: 0.1575, outputLower: 0.1575 }], }, ], ]), ], ]), }); candidateCampaigns = [ new FakeCampaign("1", {}, { format: "-1" }), new FakeCampaign("2", {}, { format: "-1" }), new FakeCampaign("3", {}, { format: "-1" }), new FakeCampaign("4", {}, { format: "-1" }), new FakeCampaign("5", {}, { format: "-1" }), new FakeCampaign("6", {}, { format: "-1" }), new FakeCampaign("7", {}, { format: "-1" }), new FakeCampaign("8", {}, { format: "-1" }), ]; campaignData = [ { id: "1", creatives: [{ id: 11, name: "cr11", format: { id: "format", ad_type: "display" } }], economic_type: "cpc", budget: 10, clicks_sold: 100, }, { id: "2", creatives: [{ id: 21, name: "cr21", format: { id: "format", ad_type: "display" } }], economic_type: "cpc", budget: 20, clicks_sold: 200, }, { id: "3", creatives: [{ id: 31, name: "cr31", format: { id: "format", ad_type: "display" } }], economic_type: "cpc", budget: 30, clicks_sold: 300, }, { id: "4", creatives: [{ id: 41, name: "cr41", format: { id: "format", ad_type: "display" } }], economic_type: "cpc", budget: 40, clicks_sold: 400, }, { id: "5", creatives: [{ id: 51, name: "cr51", format: { id: "format", ad_type: "display" } }], economic_type: "cpc", budget: 50, clicks_sold: 500, }, { id: "6", creatives: [{ id: 61, name: "cr61", format: { id: "format", ad_type: "display" } }], economic_type: "cpc", budget: 60, clicks_sold: 600, }, { id: "7", creatives: [{ id: 71, name: "cr71", format: { id: "format", ad_type: "display" } }], economic_type: "cpc", budget: 70, clicks_sold: 700, }, { id: "8", creatives: [{ id: 81, name: "cr81", format: { id: "format", ad_type: "display" } }], economic_type: "cpc", budget: 80, clicks_sold: 800, }, ]; featureConfigService = FeatureConfigService.getInstance(); mockedFeatureConf = new MockedFeatureConfig(); featureConfigService.setFeatureConfig(mockedFeatureConf); speedService = new SpeedService(); speedService.addThresholdMetadata = (request) => { for (const campaign of request.campaigns) { campaign.metadata.speed_threshold = parseInt(campaign.id) / 10; campaign.metadata.delay = parseInt(campaign.id) / 10; } }; orderingStep = new OrderingStep( campaignModel, contextService, speedService, audienceHashMap, callToActionHashMap ); }); it("campaign scores internal - no crea candidate - should take the first crea", async function () { const stepRes = await orderingStep.execute( new FakeRequestBuilder() .withCampaigns(...candidateCampaigns) .withCampaignData(campaignData) .withAppBundleOrDomain("an.app") .withSSP("ogury") .withAdUnitId("aui1") .build() ); checkMetadata(stepRes.passedCampaigns, "1", { speed_threshold: 0.1, delay: 0.1, context_score: 0.1, cta_score: null, audience_score: 0.1, composite_score: 1, }); checkMetadata(stepRes.passedCampaigns, "2", { speed_threshold: 0.2, delay: 0.2, context_score: 0.2, cta_score: null, audience_score: 0.1, composite_score: 2, }); checkMetadata(stepRes.passedCampaigns, "3", { speed_threshold: 0.3, delay: 0.3, context_score: 0.3, cta_score: null, audience_score: 0.1, composite_score: 3, }); checkMetadata(stepRes.passedCampaigns, "4", { speed_threshold: 0.4, delay: 0.4, context_score: 0.4, cta_score: null, audience_score: 0.1, composite_score: 4, }); checkMetadata(stepRes.passedCampaigns, "5", { speed_threshold: 0.5, delay: 0.5, context_score: 0.5, cta_score: 0.25, audience_score: 0.2, composite_score: 5, }); checkMetadata(stepRes.passedCampaigns, "6", { speed_threshold: 0.6, delay: 0.6, context_score: 0.6, cta_score: 0.3, audience_score: 0.2, composite_score: 6, }); checkMetadata(stepRes.passedCampaigns, "7", { speed_threshold: 0.7, delay: 0.7, context_score: 0.7, cta_score: 0.35, audience_score: 0.2, composite_score: 7, }); checkMetadata(stepRes.passedCampaigns, "8", { speed_threshold: 0.8, delay: 0.8, context_score: 0.8, cta_score: 0.4, audience_score: 0.2, composite_score: 8, }); const ids: string[] = stepRes.passedCampaigns.map((c) => c.id); expect(ids.slice(0, 4)).to.have.ordered.members(["8", "7", "6", "5"]); expect(ids.slice(4, 8)).to.have.members(["1", "2", "3", "4"]); }); describe("getCompositeScore", function () { describe("default strategy is normal", function () { it("should return good value if all scores are non-null", function () { const weights: Weights = { cta: 0.4, audience: 0.2, }; expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.NORMAL, weights, 0.1, 0.3, 0.5) ).to.be.closeTo(0.3, 1e-5); }); }); describe("random strategy", function () { it("should return random score (1)", function () { const randomStub = sinon.stub(Math, "random").returns(1); const weights: Weights = { cta: 0.1, audience: 0.2, }; expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.RANDOM, weights, 0.5, 0.7, 0.6) ).to.be.closeTo(1, 1e-5); randomStub.restore(); }); }); describe("normal strategy", function () { it("return 0 if all scores are null or zero-weighted", function () { const weights: Weights = { cta: 0, audience: 0, }; expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.NORMAL, weights, null, 0.7, 0.7) ).to.be.closeTo(0, 1e-5); weights.audience = 1; weights.cta = 0; expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.NORMAL, weights, 0.7, null, 0.7) ).to.be.closeTo(0, 1e-5); weights.audience = 0; weights.cta = 1; expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.NORMAL, weights, 0.7, 0.5, null) ).to.be.closeTo(0, 1e-5); weights.audience = 0.2; weights.cta = 0; expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.NORMAL, weights, null, null, 0.7) ).to.be.closeTo(0, 1e-5); weights.audience = 0; weights.cta = 0.1; expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.NORMAL, weights, null, 0.7, null) ).to.be.closeTo(0, 1e-5); weights.audience = 0.2; weights.cta = 0.8; expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.NORMAL, weights, 0.7, null, null) ).to.be.closeTo(0, 1e-5); weights.audience = 0.2; weights.cta = 0.1; expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.NORMAL, weights, null, null, null) ).to.be.closeTo(0, 1e-5); }); it("should ignore null scores, and redistribute their weight to other scores", function () { const weights: Weights = { cta: 0, audience: 0, }; weights.audience = 0.2; weights.cta = 0.1; expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.NORMAL, weights, null, 0.7, 0.7) ).to.be.closeTo(0.7, 1e-5); expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.NORMAL, weights, 0.7, null, 0.7) ).to.be.closeTo(0.7, 1e-5); weights.audience = 0.1; weights.cta = 0.2; expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.NORMAL, weights, 0.7, 0.7, null) ).to.be.closeTo(0.7, 1e-5); weights.audience = 0.2; weights.cta = 0.1; expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.NORMAL, weights, null, null, 0.7) ).to.be.closeTo(0.7, 1e-5); expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.NORMAL, weights, null, 0.7, null) ).to.be.closeTo(0.7, 1e-5); expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.NORMAL, weights, 0.7, null, null) ).to.be.closeTo(0.7, 1e-5); }); it("should return good value if all scores are non-null", function () { const weights: Weights = { cta: 0, audience: 0, }; weights.audience = 0.2; weights.cta = 0.4; expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.NORMAL, weights, 0.1, 0.3, 0.5) ).to.be.closeTo(0.3, 1e-5); weights.audience = 0.6; weights.cta = 0.4; expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.NORMAL, weights, 0.1, 0.3, 0.5) ).to.be.closeTo(0.38, 1e-5); weights.audience = 0; weights.cta = 0.4; expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.NORMAL, weights, 0.1, 0.3, 0.5) ).to.be.closeTo(0.26, 1e-5); weights.audience = 0.2; weights.cta = 0; expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.NORMAL, weights, 0.1, 0.3, 0.5) ).to.be.closeTo(0.14, 1e-5); weights.audience = 0; weights.cta = 1; expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.NORMAL, weights, 0.1, 0.3, 0.5) ).to.be.closeTo(0.5, 1e-5); weights.audience = 1; weights.cta = 0; expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.NORMAL, weights, 0.1, 0.3, 0.5) ).to.be.closeTo(0.3, 1e-5); weights.audience = 0; weights.cta = 0; expect( OrderingStep.getCompositeScore(CompositeScoreStrategy.NORMAL, weights, 0.1, 0.3, 0.5) ).to.be.closeTo(0.1, 1e-5); }); }); }); describe("normalizeScores", function () { let request: ChoiceRequest; let campaignCreatives: CampaignCreative[]; beforeEach(() => { const campaign = new FakeCampaign( "1", { context_score: 0.4, audience_score: 0.5, cta_score: 0.3, }, { accMetadata: { bucketsMap: [{ inputUpper: 4, outputUpper: 4, outputLower: 4 }] }, audienceMetadata: { bucketsMap: [{ inputUpper: 5, outputUpper: 5, outputLower: 5 }], brief: "", fallback: false, }, ctaMetadata: { bucketsMap: [{ inputUpper: 3, outputUpper: 3, outputLower: 3 }] }, } ); request = new FakeRequestBuilder() .withAppBundleOrDomain("an.app") .withCampaigns(campaign) .withCampaignData(campaignData) .build(); campaignCreatives = [ { campaign: campaign, creativeId: 0, metadata: Object.assign({}, campaign.metadata), metadataTmp: Object.assign({}, campaign.metadataTmp), }, ]; }); it("should return good value weights are not set", function () { orderingStep.uniformizeScores(request, campaignCreatives); expect(campaignCreatives[0].metadata.composite_score).to.equal(4.0); }); it("should return good value when composition is set to sum, and weights are set but invalid", function () { mockedFeatureConf.setParam( constants.COMPOSITE_SCORE_WEIGHTS_PARAM, `{"default":{"audience":0.2,"cta":0.3}` ); orderingStep.uniformizeScores(request, campaignCreatives); expect(campaignCreatives[0].metadata.composite_score).to.equal(4.0); }); it("should return good value when weights are set", async function () { mockedFeatureConf.setParam( constants.COMPOSITE_SCORE_WEIGHTS_PARAM, `{"default":{"audience":0.2,"cta":0.3}}` ); orderingStep.uniformizeScores(request, campaignCreatives); expect(campaignCreatives[0].metadata.composite_score).to.equal(4 * 0.5 + 5 * 0.2 + 3 * 0.3); }); it("should return cpm specific values when composition cpm weights are defined", function () { mockedFeatureConf.setParam( constants.COMPOSITE_SCORE_WEIGHTS_PARAM, `{"default":{"audience":0.15,"cta":0.5},"cpm":{"audience":0.15,"cta":0.85}}` ); campaignCreatives[0].metadataTmp.economicType = "cpm"; campaignCreatives[0].metadataTmp.adType = "video"; orderingStep.uniformizeScores(request, campaignCreatives); expect(campaignCreatives[0].metadata.composite_score).be.approximately( 0 + 5 * 0.15 + 3 * 0.85, 0.000001 ); }); it("should use adType+ecoType specific weights when defined", function () { mockedFeatureConf.setParam( constants.COMPOSITE_SCORE_WEIGHTS_PARAM, `{"default":{"audience":0.15,"cta":0.5},"cpm-video":{"audience":0.15,"cta":0.75}}` ); campaignCreatives[0].metadataTmp.economicType = "cpm"; campaignCreatives[0].metadataTmp.adType = "video"; orderingStep.uniformizeScores(request, campaignCreatives); expect(campaignCreatives[0].metadata.composite_score).to.approximately( 4 * 0.1 + 5 * 0.15 + 3 * 0.75, 0.000001 ); }); it("should use cpm weights if defined when not matching adType+ecoType specifics", function () { mockedFeatureConf.setParam( constants.COMPOSITE_SCORE_WEIGHTS_PARAM, `{"default":{"audience":0.15,"cta":0.5},"cpm":{"audience":0.15,"cta":0.65},"cpm-video":{"audience":0.15,"cta":0.85}}` ); campaignCreatives[0].metadataTmp.economicType = "cpm"; campaignCreatives[0].metadataTmp.adType = "display"; orderingStep.uniformizeScores(request, campaignCreatives); expect(campaignCreatives[0].metadata.composite_score).to.approximately( 4 * 0.2 + 5 * 0.15 + 3 * 0.65, 0.000001 ); }); }); describe("uniformize", function () { it("should return the correct value", function () { const randomStub = sinon.stub(Math, "random").returns(0.5); expect(OrderingStep.uniformize(null, null)).to.be.equal(null); expect(OrderingStep.uniformize(null, [])).to.be.equal(null); expect(OrderingStep.uniformize(2, null)).to.be.equal(0.5); expect( OrderingStep.uniformize(4.5, [{ inputUpper: 4.5, outputUpper: 0, outputLower: 1 }]) ).to.be.equal(0.5); randomStub.restore(); }); it("bisectRight should work correctly", function () { // With one item expect(bisectRight([1.0], -0.01)).to.be.equal(0); expect(bisectRight([1.0], 0.0)).to.be.equal(0); expect(bisectRight([1.0], 0.99)).to.be.equal(0); expect(bisectRight([1.0], 1.0)).to.be.equal(1); expect(bisectRight([1.0], 1.01)).to.be.equal(1); // With multiple items expect(bisectRight([0.1, 0.45, 1.0], -0.01)).to.be.equal(0); expect(bisectRight([0.1, 0.45, 1.0], 0.0)).to.be.equal(0); expect(bisectRight([0.1, 0.45, 1.0], 0.09)).to.be.equal(0); expect(bisectRight([0.1, 0.45, 1.0], 0.1)).to.be.equal(1); expect(bisectRight([0.1, 0.45, 1.0], 0.11)).to.be.equal(1); expect(bisectRight([0.1, 0.45, 1.0], 0.44)).to.be.equal(1); expect(bisectRight([0.1, 0.45, 1.0], 0.45)).to.be.equal(2); expect(bisectRight([0.1, 0.45, 1.0], 0.46)).to.be.equal(2); expect(bisectRight([0.1, 0.45, 1.0], 0.99)).to.be.equal(2); expect(bisectRight([0.1, 0.45, 1.0], 1)).to.be.equal(3); expect(bisectRight([0.1, 0.45, 1.0], 1.01)).to.be.equal(3); }); describe("should return scores with close to expected distribution", function () { const numDraws = 1000000; const numBins = 100; const binRange = [...Array(numBins).keys()]; const relativeTolerance = 5 / 100; // max 5% relative error function randomScore(low: number = 0, high: number = 1) { return Math.random() * (high - low) + low; } function runSimulation(opts: { buckets: BucketMapping[] | undefined; inputLow: number; inputHigh: number; }): number[] { const outputHist: number[] = Array(numBins).fill(0); for (let i = 0; i < numDraws; i++) { const inputScore = randomScore(opts.inputLow, opts.inputHigh); const outputScore = OrderingStep.uniformize(inputScore, opts.buckets); const outputBin = Math.floor(outputScore! * numBins); outputHist[outputBin] += 1; } return outputHist; } function assertDistributionsCloseness(outputHist: number[], expectedHist: number[]) { for (let i = 0; i < numBins; i++) { expect(outputHist[i]).to.be.closeTo(expectedHist[i], relativeTolerance * expectedHist[i]); } } it("when bucketsMap is undefined", function () { const outputHist = runSimulation({ buckets: undefined, inputLow: 0.31, inputHigh: 0.76, }); // We expect uniform distribution between 0 and 1 const expectedHist = binRange.map((_) => numDraws / numBins); assertDistributionsCloseness(outputHist, expectedHist); }); it("when bucketsMap is empty", function () { const outputHist = runSimulation({ buckets: [], inputLow: 0.25, inputHigh: 0.56, }); // We expect uniform distribution between 0 and 1 const expectedHist = binRange.map((_) => numDraws / numBins); assertDistributionsCloseness(outputHist, expectedHist); }); it("when bucketsMap has a single bucket", function () { const outputHist = runSimulation({ buckets: [ { inputUpper: 1, outputLower: 0, outputUpper: 1, }, ], inputLow: 0.21, inputHigh: 0.6, }); // We expect uniform distribution between 0 and 1 const expectedHist = binRange.map((_) => numDraws / numBins); assertDistributionsCloseness(outputHist, expectedHist); }); it("when bucketsMap has two buckets", function () { const buckets = [ { inputUpper: 0.4, outputLower: 0, outputUpper: 0.6, }, { inputUpper: 1, outputLower: 0.6, outputUpper: 1, }, ]; // Fill first bucket only const outputHist1 = runSimulation({ buckets: buckets, inputLow: 0, inputHigh: 0.4, }); // We expect uniform distribution between 0 and 0.6 const expectedHist1 = binRange.map((i) => i < 0.6 * numBins ? numDraws / numBins / (0.6 - 0) : 0 ); assertDistributionsCloseness(outputHist1, expectedHist1); // Fill second bucket only const outputHist2 = runSimulation({ buckets: buckets, inputLow: 0.4, inputHigh: 1, }); // We expect uniform distribution between 0.6 and 1 const expectedHist2 = binRange.map((i) => i >= 0.6 * numBins ? numDraws / numBins / (1 - 0.6) : 0 ); assertDistributionsCloseness(outputHist2, expectedHist2); }); it("when bucketsMap has three buckets", function () { const buckets = [ { inputUpper: 0.33, outputLower: 0, outputUpper: 0.67, }, { inputUpper: 0.55, outputLower: 0.67, outputUpper: 0.98, }, { inputUpper: 1, outputLower: 0.98, outputUpper: 1, }, ]; // Fill first bucket only const outputHist1 = runSimulation({ buckets: buckets, inputLow: 0, inputHigh: 0.33, }); // We expect uniform distribution between 0 and 0.67 const expectedHist1 = binRange.map((i) => i < 0.67 * numBins ? numDraws / numBins / (0.67 - 0) : 0 ); assertDistributionsCloseness(outputHist1, expectedHist1); // Fill second bucket only const outputHist2 = runSimulation({ buckets: buckets, inputLow: 0.33, inputHigh: 0.55, }); // We expect uniform distribution between 0.67 and 0.98 const expectedHist2 = binRange.map((i) => i >= 0.67 * numBins && i < 0.98 * numBins ? numDraws / numBins / (0.98 - 0.67) : 0 ); assertDistributionsCloseness(outputHist2, expectedHist2); // Fill third bucket only const outputHist3 = runSimulation({ buckets: buckets, inputLow: 0.55, inputHigh: 1, }); // We expect uniform distribution between 0.98 and 1 const expectedHist3 = binRange.map((i) => i >= 0.98 * numBins ? numDraws / numBins / (1 - 0.98) : 0 ); assertDistributionsCloseness(outputHist3, expectedHist3); }); }); }); describe("get campaign features", function () { it("should test getCampaignFeaturesForOrdering", async () => { const expectedObject = { "1": { economicType: "cpc", format: "SBA", budget: null, managed: 1, advertiserId: "-1", adType: "t1", }, "2": { economicType: "cpv", format: "MPU", budget: null, managed: 1, advertiserId: "-1", adType: "t1", }, "3": { economicType: "cpm", format: "INT", budget: null, managed: 1, advertiserId: "-1", adType: "t1", }, "4": { economicType: "cpe", format: "-1", budget: null, managed: 1, advertiserId: "-1", adType: "t1", }, "5": { economicType: "cpm", format: "-1", budget: 3000, managed: 1, advertiserId: "-1", adType: "t1", }, }; const testCampaigns: Record = { 1: { id: "1", economic_type: "cpc", creatives: [{ id: 11, name: "cr11", format: { id: "204", ad_type: "t1" } }], environment_type: "in-app", }, 2: { id: "2", economic_type: "cpv", creatives: [{ id: 12, name: "cr11", format: { id: "205", ad_type: "t1" } }], environment_type: "in-app", }, 3: { id: "3", economic_type: "cpm", creatives: [{ id: 13, name: "cr11", format: { id: "160", ad_type: "t1" } }], environment_type: "in-app", }, 4: { id: "4", economic_type: "cpe", creatives: [{ id: 14, name: "cr11", format: { id: "1111", ad_type: "t1" } }], environment_type: "in-app", }, 5: { id: "5", economic_type: "cpm", creatives: [{ id: 14, name: "cr11", format: { id: "1111", ad_type: "t1" } }], environment_type: "in-app", budget: 3000, }, }; const resultObj: any = {}; const campaignCreatives: CampaignCreative[] = Object.keys(testCampaigns).map((key) => { return { campaign: new FakeCampaign(key), creativeId: testCampaigns[key].creatives![0].id, metadata: {}, metadataTmp: {}, }; }); const request = new FakeRequestBuilder().build(); request.campaignData = testCampaigns; await Promise.all( campaignCreatives.map((cc) => { resultObj[cc.campaign.id] = orderingStep.getCampaignFeatures(request, cc); }) ); expect(resultObj).to.deep.equal(expectedObject); }); it("should test getCampaignFeaturesForOrdering (multicrea)", async () => { const testCampaigns: Record = { 1: { id: "1", economic_type: "cpc", environment_type: "in-app", creatives: [ { id: 11, name: "creative 11", format: { id: "204", ad_type: "t1", }, }, ], }, 2: { id: "2", economic_type: "cpv", environment_type: "in-app", creatives: [ { id: 21, name: "creative 21", format: { id: "204", ad_type: "t1", }, }, { id: 22, name: "creative 22", format: { id: "205", ad_type: "t2", }, }, ], }, 3: { id: "3", economic_type: "cpm", environment_type: "in-app", creatives: [ { id: 31, name: "creative 31", format: { id: "160", ad_type: "t1", }, }, { id: 32, name: "creative 32", format: { id: "204", ad_type: "t3", }, }, ], }, }; // test with first crea let campaignCreatives: CampaignCreative[] = Object.keys(testCampaigns).map((key) => { return { campaign: new FakeCampaign(key), creativeId: testCampaigns[key].creatives![0].id, metadata: {}, metadataTmp: {}, }; }); let request = new FakeRequestBuilder().build(); request.campaignData = testCampaigns; let campaignFeatures = campaignCreatives.map((cc) => orderingStep.getCampaignFeatures(request, cc) ); let expectedObject: any[] = [ { economicType: "cpc", format: "SBA", budget: null, managed: 1, advertiserId: "-1", adType: "t1", }, { economicType: "cpv", format: "SBA", budget: null, managed: 1, advertiserId: "-1", adType: "t1", }, { economicType: "cpm", format: "INT", budget: null, managed: 1, advertiserId: "-1", adType: "t1", }, ]; expect(campaignFeatures).to.deep.equal(expectedObject); // test with last crea campaignCreatives = Object.keys(testCampaigns).map((key) => { return { campaign: new FakeCampaign(key), creativeId: testCampaigns[key].creatives![testCampaigns[key].creatives!.length - 1].id, metadata: {}, metadataTmp: {}, }; }); request = new FakeRequestBuilder().build(); request.campaignData = testCampaigns; campaignFeatures = campaignCreatives.map((cc) => orderingStep.getCampaignFeatures(request, cc) ); expectedObject = [ { economicType: "cpc", format: "SBA", budget: null, managed: 1, advertiserId: "-1", adType: "t1", }, { economicType: "cpv", format: "MPU", budget: null, managed: 1, advertiserId: "-1", adType: "t2", }, { economicType: "cpm", format: "SBA", budget: null, managed: 1, advertiserId: "-1", adType: "t3", }, ]; expect(campaignFeatures).to.deep.equal(expectedObject); }); }); describe("Campaigns are ranked", function () { it("highest income campaigns first", async function () { const candidates = [ { id: "1", metadata: { acc_rate: 0.05 } }, { id: "2", metadata: { acc_rate: 0.01 } }, { id: "3", metadata: { acc_rate: 0.1 } }, { id: "4", metadata: { acc_rate: 0.02 } }, ]; const campaignData = candidates.map((c) => { return { id: c.id, creatives: [{} as any], clicks_sold: 1000, budget: 20, }; }); sinon.stub(contextService, "addScores"); const stepRes = await orderingStep.execute( new FakeRequestBuilder() .withCampaigns( ...candidates.map( (c) => new FakeCampaign(c.id, c.metadata, { accMetadata: { bucketsMap: [] }, }) ) ) .withCampaignData(campaignData) .build() ); const ids: string[] = stepRes.passedCampaigns.map((c) => c.id); expect(ids).to.have.ordered.members(["3", "1", "4", "2"]); }); }); }); describe("Ordering step - multiple creatives", function () { let campaignModel: CampaignModelFake; let contextService; let callToActionHashMap: CallToActionHashMapFake; let audienceHashMap: AudienceHashMapFake; let candidateCampaigns: Campaign[]; let campaignData: Partial[]; let featureConfigService: FeatureConfigService; let mockedFeatureConf: MockedFeatureConfig; let speedService: SpeedService; let orderingStep: OrderingStep; before(() => { campaignModel = new CampaignModelFake(); // these values will be possible after dimension 'creative' is added to accomplish rate segment contextService = new ContextServiceFake({ "1": { "11": { score: 0.1, bucketsMap: [{ inputUpper: 1, outputUpper: 1, outputLower: 1 }] }, }, "2": { "21": { score: 0.2, bucketsMap: [{ inputUpper: 2, outputUpper: 2, outputLower: 2 }] }, }, "3": { "31": { score: 0.3, bucketsMap: [{ inputUpper: 3, outputUpper: 3, outputLower: 3 }] }, }, "4": { "41": { score: 0.41, bucketsMap: [{ inputUpper: 4, outputUpper: 4, outputLower: 4 }] }, "42": { score: 0.42, bucketsMap: [{ inputUpper: 4, outputUpper: 4, outputLower: 4 }] }, }, "5": { "51": { score: 0.51, bucketsMap: [{ inputUpper: 5.1, outputUpper: 5.1, outputLower: 5.1 }], }, "52": { score: 0.52, bucketsMap: [{ inputUpper: 5.2, outputUpper: 5.2, outputLower: 5.2 }], }, }, "6": { "61": { score: 0.61, bucketsMap: [{ inputUpper: 6.1, outputUpper: 6.1, outputLower: 6.1 }], }, "62": { score: 0.62, bucketsMap: [{ inputUpper: 6.2, outputUpper: 6.2, outputLower: 6.2 }], }, }, "7": { "71": { score: 0.71, bucketsMap: [{ inputUpper: 7.1, outputUpper: 7.1, outputLower: 7.1 }], }, "72": { score: 0.72, bucketsMap: [{ inputUpper: 7.2, outputUpper: 7.2, outputLower: 7.2 }], }, "73": { score: 0.73, bucketsMap: [{ inputUpper: 7.3, outputUpper: 7.3, outputLower: 7.3 }], }, }, "8": { "81": { score: 0.85, bucketsMap: [{ inputUpper: 8.5, outputUpper: 8.5, outputLower: 8.5 }], }, "82": { score: 0.82, bucketsMap: [{ inputUpper: 8.2, outputUpper: 8.2, outputLower: 8.2 }], }, "83": { score: 0.83, bucketsMap: [{ inputUpper: 8.3, outputUpper: 8.3, outputLower: 8.3 }], }, }, }); callToActionHashMap = CallToActionHashMapFake.fromCtaMaps({ scoresMap: new Map([ ["-1|ogury|5|an.app|aui1", 0.25], ["-1|ogury|6|-1|-1", 0.3], ["-1|adru|6|-1|-1", 0.3], ["-1|ogury|7|an.app|aui1", 0.35], ["-1|ogury|8|-1|-1", 0.4], ["-1|adru|8|-1|-1", 0.4], ["MPU|-1|-1|-1|-1", 0.5], ]), normMap: new Map([ [ "-1|1", { bucketsMap: [ { inputUpper: 0.1, outputLower: 0, outputUpper: 0.76, }, { inputUpper: 0.785, outputLower: 0.76, outputUpper: 0.91, }, { inputUpper: 1, outputLower: 0.91, outputUpper: 1, }, ], }, ], [ "-1|2", { bucketsMap: [ { inputUpper: 0.1, outputLower: 0, outputUpper: 0.76, }, { inputUpper: 0.785, outputLower: 0.76, outputUpper: 0.92, }, { inputUpper: 1, outputLower: 0.92, outputUpper: 1, }, ], }, ], [ "MPU|-1", { bucketsMap: [ { inputUpper: 0.1, outputLower: 0, outputUpper: 0.76, }, { inputUpper: 0.785, outputLower: 0.76, outputUpper: 0.93, }, { inputUpper: 1, outputLower: 0.93, outputUpper: 1, }, ], }, ], [ "-1|-1", { bucketsMap: [ { inputUpper: 0.1, outputLower: 0, outputUpper: 0.76, }, { inputUpper: 0.785, outputLower: 0.76, outputUpper: 0.94, }, { inputUpper: 1, outputLower: 0.94, outputUpper: 1, }, ], }, ], ]), }); audienceHashMap = AudienceHashMapFake.fromAudienceScores({ apps: new Map([ ["an.app", 0], ["appBundle", 1], ]), campaigns: new Map([ ["1", "0"], ["2", "0"], ["3", "0"], ["4", "0"], ["5", "1"], ["6", "1"], ["7", "1"], ["8", "1"], ]), similarities: new Map([ [ "0", new Map([ [ 0, { score: 0.1, bucketsMap: [{ inputUpper: 0.1, outputUpper: 0.1, outputLower: 0.1 }] }, ], [ 1, { score: 0.11, bucketsMap: [{ inputUpper: 0.11, outputUpper: 0.11, outputLower: 0.11 }], }, ], [ -1, { score: 0.105, bucketsMap: [{ inputUpper: 0.105, outputUpper: 0.105, outputLower: 0.105 }], }, ], ]), ], [ "1", new Map([ [ 0, { score: 0.2, bucketsMap: [{ inputUpper: 0.2, outputUpper: 0.2, outputLower: 0.2 }] }, ], [ 1, { score: 0.22, bucketsMap: [{ inputUpper: 0.22, outputUpper: 0.22, outputLower: 0.22 }], }, ], [ -1, { score: 0.21, bucketsMap: [{ inputUpper: 0.21, outputUpper: 0.21, outputLower: 0.21 }], }, ], ]), ], [ "-1", new Map([ [ -1, { score: 0.1575, bucketsMap: [{ inputUpper: 0.1575, outputUpper: 0.1575, outputLower: 0.1575 }], }, ], ]), ], ]), }); candidateCampaigns = [ new FakeCampaign("1", undefined, undefined, [11]), new FakeCampaign("2", undefined, undefined, [21]), new FakeCampaign("3", undefined, undefined, [31]), new FakeCampaign("4", undefined, undefined, [41, 42]), new FakeCampaign("5", undefined, undefined, [51, 52]), new FakeCampaign("6", undefined, undefined, [61, 62]), new FakeCampaign("7", undefined, undefined, [71, 72, 73]), new FakeCampaign("8", undefined, undefined, [81, 82, 83]), ]; campaignData = [ { id: "1", economic_type: "cpc", creatives: [{ id: 11, name: "creative 11", format: { id: "format", ad_type: "display" } }], budget: 10, clicks_sold: 100, }, { id: "2", economic_type: "cpc", creatives: [{ id: 21, name: "creative 21", format: { id: "format", ad_type: "display" } }], budget: 20, clicks_sold: 200, }, { id: "3", economic_type: "cpc", creatives: [{ id: 31, name: "creative 31", format: { id: "format", ad_type: "display" } }], budget: 30, clicks_sold: 300, }, { id: "4", economic_type: "cpc", creatives: [ { id: 41, name: "creative 41", format: { id: "format", ad_type: "display" } }, { id: 42, name: "creative 42", format: { id: "format", ad_type: "display" } }, ], budget: 40, clicks_sold: 400, }, { id: "5", economic_type: "cpc", creatives: [ { id: 51, name: "creative 51", format: { id: "format", ad_type: "display" } }, { id: 51, name: "creative 52", format: { id: "format", ad_type: "display" } }, ], budget: 50, clicks_sold: 500, }, { id: "6", economic_type: "cpc", creatives: [ { id: 61, name: "creative 61", format: { id: "format", ad_type: "display" } }, { id: 62, name: "creative 62", format: { id: "format", ad_type: "display" } }, ], budget: 60, clicks_sold: 600, }, { id: "7", economic_type: "cpc", creatives: [ { id: 71, name: "creative 71", format: { id: "format", ad_type: "display" } }, { id: 72, name: "creative 72", format: { id: "format", ad_type: "display" } }, { id: 73, name: "creative 73", format: { id: "format", ad_type: "display" } }, ], budget: 70, clicks_sold: 700, }, { id: "8", economic_type: "cpc", creatives: [ { id: 81, name: "creative 81", format: { id: "format", ad_type: "display" } }, { id: 82, name: "creative 82", format: { id: "format", ad_type: "display" } }, { id: 83, name: "creative 83", format: { id: "format", ad_type: "display" } }, ], budget: 80, clicks_sold: 800, }, ]; mockedFeatureConf = new MockedFeatureConfig(); featureConfigService = FeatureConfigService.getInstance(); featureConfigService.setFeatureConfig(mockedFeatureConf); mockedFeatureConf.setParam(constants.COMPOSITE_SCORE_WEIGHTS_PARAM, ""); speedService = new SpeedService(); speedService.addThresholdMetadata = (request) => { for (const campaign of request.campaigns) { campaign.metadata.speed_threshold = parseInt(campaign.id) / 10; campaign.metadata.delay = parseInt(campaign.id) / 10; } }; orderingStep = new OrderingStep( campaignModel, contextService, speedService, audienceHashMap, callToActionHashMap ); }); it("campaign scores internal - multiple creatives", async function () { // first creative is selected sinon.stub(helper, "getRandomItem").callsFake((array) => array[0]); const stepRes = await orderingStep.execute( new FakeRequestBuilder() .withCampaigns(...candidateCampaigns) .withCampaignData(campaignData) .withAppBundleOrDomain("an.app") .withSSP("ogury") .withAdUnitId("aui1") .build() ); checkMetadata(stepRes.passedCampaigns, "1", { speed_threshold: 0.1, delay: 0.1, context_score: 0.1, cta_score: null, audience_score: 0.1, composite_score: 1, creative_id: 11, }); checkMetadata(stepRes.passedCampaigns, "2", { speed_threshold: 0.2, delay: 0.2, context_score: 0.2, cta_score: null, audience_score: 0.1, composite_score: 2, creative_id: 21, }); checkMetadata(stepRes.passedCampaigns, "3", { speed_threshold: 0.3, delay: 0.3, context_score: 0.3, cta_score: null, audience_score: 0.1, composite_score: 3, creative_id: 31, }); checkMetadata(stepRes.passedCampaigns, "4", { speed_threshold: 0.4, delay: 0.4, context_score: 0.41, cta_score: null, audience_score: 0.1, composite_score: 4, creative_id: 41, }); checkMetadata(stepRes.passedCampaigns, "5", { speed_threshold: 0.5, delay: 0.5, context_score: 0.51, cta_score: 0.25, audience_score: 0.2, composite_score: 5.1, creative_id: 51, }); checkMetadata(stepRes.passedCampaigns, "6", { speed_threshold: 0.6, delay: 0.6, context_score: 0.61, cta_score: 0.3, audience_score: 0.2, composite_score: 6.1, creative_id: 61, }); checkMetadata(stepRes.passedCampaigns, "7", { speed_threshold: 0.7, delay: 0.7, context_score: 0.71, cta_score: 0.35, audience_score: 0.2, composite_score: 7.1, creative_id: 71, }); checkMetadata(stepRes.passedCampaigns, "8", { speed_threshold: 0.8, delay: 0.8, context_score: 0.85, cta_score: 0.4, audience_score: 0.2, composite_score: 8.5, creative_id: 81, }); const ids: string[] = stepRes.passedCampaigns.map((c) => c.id); expect(ids.slice(0, 4)).to.have.ordered.members(["8", "7", "6", "5"]); expect(ids.slice(4, 8)).to.have.members(["1", "2", "3", "4"]); }); }); function checkMetadata(campaigns: Campaign[], id: string, expectedMetadata: any) { const campaign = campaigns.find((campaign) => campaign.id === id); if (expectedMetadata.context_score !== undefined) { expectedMetadata.acc_rate = expectedMetadata.context_score; } Object.keys(expectedMetadata).forEach((key) => { expect((campaign!.metadata as any)[key]).to.deep.equal( expectedMetadata[key], `Metadata key ${key}` ); }); }