kuhyx/WUT_Computer_Science
1
0
Fork 0
mirror of https://github.com/kuhyx/WUT_Computer_Science.git synced 2026-11-10 21:27:08 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

Hold the coursework to the same lint bar as the tooling

Browse Source 
kuhy overruled the "do not fix coursework to be better than it was submitted"
rule: git history is the archive, so the working tree can be clean. This lands
the config and the mechanical half of the cleanup -- 9,281 ruff findings under
the fleet config down to 3,686, with no per-file-ignores anywhere.

What is in here:

- pyproject.toml, the first one in this repo. select = ["ALL"], mypy strict,
  and the vendored list mirrored into ruff exclude and coverage omit. Five
  rules are off and each says why; CPY001 is the only judgement call, because
  stamping a copyright header onto 137 university assignments would assert a
  licence that was never claimed. No hook is wired up yet -- a gate lands only
  once its subject is clean.

- ruff format + the safe fixes, then a second pass over the docstring rules
  ruff marks unsafe (they only edit docstring text). 108 files.

- TRAK/sightpy is vendored, not coursework: it is rafael-fuente/Python-Raytracer
  (BSD 3-Clause). Verified against a fresh upstream clone with line endings
  normalised -- 30 of 32 files byte-identical, ray.py and scene.py carrying a
  12- and 13-line local patch that counts rays for the report. 35 files that
  should never have been linted, now excluded and restored.

- WDWR, SPD, MOM and ECRYPT are fully clean, by hand and verified by running
  them: WDWR's three plot scripts print the same statistics as before, MOM's
  PuLP model solves to the same cost. WDWR's graph.py had a 342-character CSV
  header embedded as a string literal, which cannot be wrapped -- the data
  moved to data/pareto_front.csv. Two 0-byte .py stubs are deleted; an empty
  file is not a deliverable and git still has it.

Three harness bugs found by actually running things:

- `run` now forces MPLBACKEND=Agg. TRAK finished its ray trace in 37 seconds
  and then sat for 45 minutes: plt.show() had opened an image viewer and was
  waiting for someone to close the window.
- TRAK's main.py only pops that viewer with --show now, and its runner writes
  to a gitignored build/ via the new --entry-args. The default output path is
  outputs/output.png, which is TRACKED -- a plain run overwrote a submitted
  deliverable.
- course_lib.sh was 357 lines, over this repo's own cap. Split into
  course_probe.sh and course_run.sh.

Verified: ECOTE 26 tests pass, ECRYPT_PROJECT 99 pass, TRAK renders
cornell_box in 37s and exits 0, every module compiles, and every import ruff
removed was checked to be genuinely unused.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0172dj34FDpao6wsix8ZuCzE
This commit is contained in:
Krzysztof Rudnicki 2026-08-28 19:53:04 +02:00
parent ffa5dad02a
commit bd24dd57ea
93 changed files with 5367 additions and 3844 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

23
CLAUDE.md
View File

@ -71,9 +71,18 @@ check.
shared gate in `~/utils`. There is deliberately no baseline and no shared gate in `~/utils`. There is deliberately no baseline and no
allowlist: split the file. For a LaTeX report that means `\input{}` per allowlist: split the file. For a LaTeX report that means `\input{}` per
section; for a markdown report, sibling `DOCS-*.md` files behind an index. section; for a markdown report, sibling `DOCS-*.md` files behind an index.
- Do not "fix" coursework to be better than it was submitted. Correct what is - **Coursework is held to the same bar as the tooling.** It gets the full
broken, keep what is merely old-fashioned. The archive should still show `select = ["ALL"]` treatment -- annotations, docstrings, naming, the lot --
what was actually handed in. and there are no per-file-ignores carving the two course trees out. The
version that was handed in is preserved by git history, which is a better
archive than a frozen working tree; this repo's job is code that still
builds, runs and reads well. (This reverses an earlier rule that said not to
improve coursework beyond what was submitted. kuhy overruled it on
2026-08-28: "yes fix it if necessary we can always go back to worse version
by going back in git repo.")
- The one thing that is still off limits is *what was delivered*: do not add
content to a submitted `.pdf`/report that was never in it, and do not merge
the near-duplicate submission directories listed above.
### Vendored content ### Vendored content
@ -87,8 +96,12 @@ is mirrored into `~/utils/file_length/_tables.py`, `[tool.ruff] exclude` and
- `ruff` runs with `select = ["ALL"]` and mypy with `strict = true`. - `ruff` runs with `select = ["ALL"]` and mypy with `strict = true`.
- No `# noqa`, no `# type: ignore`. A pre-commit hook rejects both outright. - No `# noqa`, no `# type: ignore`. A pre-commit hook rejects both outright.
- Notebooks: outputs are stripped on commit, lint via `nbqa`. Real logic - Notebooks: `ruff` reads and formats `.ipynb` natively (cell-aware, so an
belongs in a sibling importable module — a notebook cannot be covered. import used two cells later is not reported unused), which is why there is
no `nbqa` layer. It rewrites `source` only — outputs are left byte-identical.
Nothing strips outputs on commit yet; the four notebooks here were committed
with theirs. Real logic belongs in a sibling importable module — a notebook
cannot be covered.
### LaTeX ### LaTeX

2
DOCS-runnability.md
View File

@ -51,7 +51,7 @@ run cannot clobber the local column.
| `Programming/PORR` | runnable | - | python3 /home/kuhy/WUT_Computer_Science/Programming/PORR/code/main.py | | `Programming/PORR` | runnable | - | python3 /home/kuhy/WUT_Computer_Science/Programming/PORR/code/main.py |
| `Programming/PSD` | runnable | - | starts the Kafka/Flink stack (installs packages with sudo) | | `Programming/PSD` | runnable | - | starts the Kafka/Flink stack (installs packages with sudo) |
| `Programming/psd_project` | runnable | - | mvn package | | `Programming/psd_project` | runnable | - | mvn package |
| `Programming/TRAK` | runnable | - | python3 the entry script | | `Programming/TRAK` | runnable | - | python3 /home/kuhy/WUT_Computer_Science/Programming/TRAK/main.py |
| `Programming/twm_4` | blocked | - | jupyter is on PATH but broken (bad interpreter) | | `Programming/twm_4` | blocked | - | jupyter is on PATH but broken (bad interpreter) |
| `Programming/USD` | no-code | - | README describes a MetaDrive RL assignment; no implementation was submitted | | `Programming/USD` | no-code | - | README describes a MetaDrive RL assignment; no implementation was submitted |
| `Programming/WDWR` | runnable | - | python3 the entry script | | `Programming/WDWR` | runnable | - | python3 the entry script |

150
NotProgramming/MOM/report_three/model.py Normal file → Executable file
View File

@ -1,45 +1,112 @@
import pulp #!/usr/bin/env python3
"""Bi-criteria distribution model for MOM project 3 (data set 1.6).
A PuLP transport problem over two factories, four warehouses and six
customers, minimising distribution cost while maximising customer
satisfaction. It is solved three ways: once on cost alone, then swept over
alpha in 0.0..1.0 to trace the efficient frontier, then at the five
weightings the report tabulates.
"""
import logging
import sys import sys
import pulp
logger = logging.getLogger(__name__)
logging.basicConfig(format="%(message)s", level=logging.INFO)
# alpha runs over 0.0, 0.1 ... 1.0, so eleven steps.
ALPHA_STEPS = 11
# Create a problem variable: # Create a problem variable:
model = pulp.LpProblem("Optimal_Distribution", pulp.LpMinimize) model = pulp.LpProblem("Optimal_Distribution", pulp.LpMinimize)
# Zdefiniowanie dostawcy (fabryki and magazyny) # Zdefiniowanie dostawcy (fabryki and magazyny)
fabryki = ['F1', 'F2'] fabryki = ["F1", "F2"]
magazyny = ['M1', 'M2', 'M3', 'M4'] magazyny = ["M1", "M2", "M3", "M4"]
dostawcy = fabryki + magazyny # Combining both lists dostawcy = fabryki + magazyny # Combining both lists
# Zdefiniowanie klientów # Zdefiniowanie klientów
klienci = ['K1', 'K2', 'K3', 'K4', 'K5', 'K6'] klienci = ["K1", "K2", "K3", "K4", "K5", "K6"]
koszt = { koszt = {
'F1': {'M1': 0.3, 'M2': 0.5, 'M3': 1.2, 'M4': 0.8, 'K1': 1.2, 'K2': 999, "F1": {
'K3': 1.2, 'K4': 2.0, 'K5': 999, 'K6': 1.1}, "M1": 0.3,
'F2': {'M1': 999, 'M2': 0.4, 'M3': 0.5, 'M4': 0.3, 'K1': 1.8, 'K2': 999, "M2": 0.5,
'K3': 999, 'K4': 999, 'K5': 999, 'K6': 999}, "M3": 1.2,
'M1': {'K1': 999, 'K2': 1.2, 'K3': 0.2, 'K4': 1.7, 'K5': 999, 'K6': 2.0}, "M4": 0.8,
'M2': {'K1': 1.4, 'K2': 0.3, 'K3': 1.8, 'K4': 1.3, 'K5': 0.5, 'K6': 999}, "K1": 1.2,
'M3': {'K1': 999, 'K2': 1.3, 'K3': 2.0, 'K4': 999, 'K5': 0.3, 'K6': 1.4}, "K2": 999,
'M4': {'K1': 999, 'K2': 999, 'K3': 0.4, 'K4': 2.0, 'K5': 0.5, 'K6': 1.6} "K3": 1.2,
"K4": 2.0,
"K5": 999,
"K6": 1.1,
},
"F2": {
"M1": 999,
"M2": 0.4,
"M3": 0.5,
"M4": 0.3,
"K1": 1.8,
"K2": 999,
"K3": 999,
"K4": 999,
"K5": 999,
"K6": 999,
},
"M1": {"K1": 999, "K2": 1.2, "K3": 0.2, "K4": 1.7, "K5": 999, "K6": 2.0},
"M2": {"K1": 1.4, "K2": 0.3, "K3": 1.8, "K4": 1.3, "K5": 0.5, "K6": 999},
"M3": {"K1": 999, "K2": 1.3, "K3": 2.0, "K4": 999, "K5": 0.3, "K6": 1.4},
"M4": {"K1": 999, "K2": 999, "K3": 0.4, "K4": 2.0, "K5": 0.5, "K6": 1.6},
} }
P = pulp.LpVariable.dicts("P", [(i, j) for i in dostawcy for j in klienci], cat='Binary') P = pulp.LpVariable.dicts(
"P", [(i, j) for i in dostawcy for j in klienci], cat="Binary"
)
maksymalne_zamowienie = 60 maksymalne_zamowienie = 60
poziom_satysfakcji = {'K1': 50 / maksymalne_zamowienie, 'K2': 10 / maksymalne_zamowienie, 'K3': 40 / maksymalne_zamowienie, 'K4': 35 / maksymalne_zamowienie, 'K5': 60 / maksymalne_zamowienie, 'K6': 20 / maksymalne_zamowienie} poziom_satysfakcji = {
preferencja_klienta = {'K1': ['F2'], 'K2': ['M1'], 'K3': ['M2', 'M3'], 'K4': ['F1'], 'K5': [], 'K6': ['M3', 'M4']} "K1": 50 / maksymalne_zamowienie,
suma_satysfakcji = pulp.lpSum([poziom_satysfakcji[j] * P[(i, j)] for i in dostawcy for j in klienci]) "K2": 10 / maksymalne_zamowienie,
"K3": 40 / maksymalne_zamowienie,
"K4": 35 / maksymalne_zamowienie,
"K5": 60 / maksymalne_zamowienie,
"K6": 20 / maksymalne_zamowienie,
}
preferencja_klienta = {
"K1": ["F2"],
"K2": ["M1"],
"K3": ["M2", "M3"],
"K4": ["F1"],
"K5": [],
"K6": ["M3", "M4"],
}
suma_satysfakcji = pulp.lpSum(
[poziom_satysfakcji[j] * P[(i, j)] for i in dostawcy for j in klienci]
)
# Zmienne decyzyjne # Zmienne decyzyjne
x = pulp.LpVariable.dicts("x", [(i, j) for i in dostawcy for j in klienci], lowBound=0, cat='Integer') x = pulp.LpVariable.dicts(
y = pulp.LpVariable.dicts("y", [(i, k) for i in fabryki for k in magazyny], lowBound=0, cat='Integer') "x", [(i, j) for i in dostawcy for j in klienci], lowBound=0, cat="Integer"
)
y = pulp.LpVariable.dicts(
"y", [(i, k) for i in fabryki for k in magazyny], lowBound=0, cat="Integer"
)
# Funkcje Celu # Funkcje Celu
koszt_dystrybucji = pulp.lpSum([koszt[i][j] * x[(i, j)] for i in dostawcy for j in klienci]) koszt_dystrybucji = pulp.lpSum(
koszt_magazynowania = pulp.lpSum([koszt[i][k] * y[(i, k)] for i in fabryki for k in magazyny]) [koszt[i][j] * x[(i, j)] for i in dostawcy for j in klienci]
)
koszt_magazynowania = pulp.lpSum(
[koszt[i][k] * y[(i, k)] for i in fabryki for k in magazyny]
)
alpha = 0.5 alpha = 0.5
beta = 0.5 beta = 0.5
model += alpha * (koszt_dystrybucji + koszt_magazynowania) - beta * suma_satysfakcji model += alpha * (koszt_dystrybucji + koszt_magazynowania) - beta * suma_satysfakcji
mozliwosci_fabryki = {'F1': 150, 'F2': 200} mozliwosci_fabryki = {"F1": 150, "F2": 200}
pojemnosc_magazynu = {'M1': 70, 'M2': 50 , 'M3': 100, 'M4': 40 } pojemnosc_magazynu = {"M1": 70, "M2": 50, "M3": 100, "M4": 40}
zamowienia_klientow = {'K1': 50, 'K2': 10, 'K3': 40, 'K4': 35, 'K5': 60, 'K6': 20} zamowienia_klientow = {"K1": 50, "K2": 10, "K3": 40, "K4": 35, "K5": 60, "K6": 20}
for i in fabryki: for i in fabryki:
model += pulp.lpSum([x[(i, j)] for j in klienci] + [y[(i, k)] for k in magazyny]) <= mozliwosci_fabryki[i] model += (
pulp.lpSum([x[(i, j)] for j in klienci] + [y[(i, k)] for k in magazyny])
<= mozliwosci_fabryki[i]
)
for k in magazyny: for k in magazyny:
model += pulp.lpSum([x[(k, j)] for j in klienci]) <= pojemnosc_magazynu[k] model += pulp.lpSum([x[(k, j)] for j in klienci]) <= pojemnosc_magazynu[k]
@ -49,22 +116,21 @@ for j in klienci:
model.solve() model.solve()
for v in model.variables(): for v in model.variables():
print(v.name, "=", v.varValue) logger.info("%s = %s", v.name, v.varValue)
koszt_wyniki = [] koszt_wyniki = []
zadowolenie_wyniki = [] zadowolenie_wyniki = []
wyniki_funkcji = [] wyniki_funkcji = []
maksymalny_wynik = 0; maksymalny_wynik = 0
for alpha in range(0, 11): for step in range(ALPHA_STEPS):
beta = 10 - alpha + sys.float_info.epsilon alpha = step / 10.0
alpha /= 10.0 beta = (ALPHA_STEPS - 1 - step + sys.float_info.epsilon) / 10.0
beta /= 10.0
# Update objective function # Update objective function
model.objective = alpha * koszt_dystrybucji - beta * suma_satysfakcji model.objective = alpha * koszt_dystrybucji - beta * suma_satysfakcji
# Solve the model # Solve the model
model.solve() model.solve()
print(alpha) logger.info("%s", alpha)
# Record the wyniki # Record the wyniki
calkowity_koszt = pulp.value(koszt_dystrybucji) calkowity_koszt = pulp.value(koszt_dystrybucji)
@ -72,14 +138,19 @@ for alpha in range(0, 11):
wynik_funkcji = pulp.value(alpha * koszt_dystrybucji - beta * suma_satysfakcji) wynik_funkcji = pulp.value(alpha * koszt_dystrybucji - beta * suma_satysfakcji)
koszt_wyniki.append(calkowity_koszt) koszt_wyniki.append(calkowity_koszt)
zadowolenie_wyniki.append(calkowite_zadowolenie) zadowolenie_wyniki.append(calkowite_zadowolenie)
if wynik_funkcji > maksymalny_wynik: maksymalny_wynik = max(maksymalny_wynik, wynik_funkcji)
maksymalny_wynik = wynik_funkcji
wyniki_funkcji.append(wynik_funkcji) wyniki_funkcji.append(wynik_funkcji)
print("maksymalny_wynik", maksymalny_wynik, wyniki_funkcji) logger.info("maksymalny_wynik %s %s", maksymalny_wynik, wyniki_funkcji)
# Pseudo-code, assuming model setup as previously discussed # Pseudo-code, assuming model setup as previously discussed
scemariusze = [(1.0, sys.float_info.epsilon), (0.8, 0.2), (0.5, 0.5), (0.2, 0.8), (sys.float_info.epsilon, 1.0)] scemariusze = [
(1.0, sys.float_info.epsilon),
(0.8, 0.2),
(0.5, 0.5),
(0.2, 0.8),
(sys.float_info.epsilon, 1.0),
]
wyniki = [] wyniki = []
for alpha, beta in scemariusze: for alpha, beta in scemariusze:
@ -92,7 +163,8 @@ for alpha, beta in scemariusze:
wyniki.append((alpha, beta, calkowity_koszt, calkowite_zadowolenie)) wyniki.append((alpha, beta, calkowity_koszt, calkowite_zadowolenie))
for idx, (alpha, beta, koszt, zadowolenie) in enumerate(wyniki): for idx, (alpha, beta, koszt, zadowolenie) in enumerate(wyniki):
print(f"Krok {idx+1}:") logger.info("Krok %d:", idx + 1)
print(f" koszt: {alpha}, zadowolenie: {beta}") logger.info(" koszt: %s, zadowolenie: %s", alpha, beta)
print(f" Calkowity koszt: {koszt}, zadowolenie klienta: {zadowolenie}\n") logger.info(
" Calkowity koszt: %s, zadowolenie klienta: %s\n", koszt, zadowolenie
)

0
NotProgramming/MOM/report_two/model.py
View File

201
NotProgramming/SPD/lab4_spd/main.py Normal file → Executable file
View File

@ -1,128 +1,167 @@
#!/usr/bin/env python3
"""Plot the SPD lab-4 masking thresholds for the 200 Hz and 400 Hz tone complexes.
Each dataset is one measured masking curve -- probe frequency against the
level at which it became audible -- and the two figures overlay every curve
recorded at that fundamental.
"""
import matplotlib.pyplot as plt import matplotlib.pyplot as plt
# Prepare the datasets # Prepare the datasets
datasets_200Hz = [] DATASETS_200_HZ = []
datasets_400Hz = [] DATASETS_400_HZ = []
# Dataset 1: Wieloton 200 Hz, 7 (1400 Hz), 3 składowe # Dataset 1: Wieloton 200 Hz, 7 (1400 Hz), 3 składowe
# Starting x value is 1400 Hz # Starting x value is 1400 Hz
datasets_200Hz.append({ DATASETS_200_HZ.append(
'name': '200 Hz, 7 (1400 Hz), 3 składowe', {
'x': [1400 + 0, 1400 + 50, 1400 + 100, 1400 + 150], "name": "200 Hz, 7 (1400 Hz), 3 składowe",
'y': [200, 200, 215, 230] "x": [1400 + 0, 1400 + 50, 1400 + 100, 1400 + 150],
}) "y": [200, 200, 215, 230],
}
)
# Dataset 2: Wieloton 200 Hz, 7 (1400 Hz), 7 składowych # Dataset 2: Wieloton 200 Hz, 7 (1400 Hz), 7 składowych
datasets_200Hz.append({ DATASETS_200_HZ.append(
'name': '200 Hz, 7 (1400 Hz), 7 składowych', {
'x': [1400 + 0, 1400 + 50, 1400 + 100, 1400 + 150], "name": "200 Hz, 7 (1400 Hz), 7 składowych",
'y': [190, 190, 200, 195] "x": [1400 + 0, 1400 + 50, 1400 + 100, 1400 + 150],
}) "y": [190, 190, 200, 195],
}
)
# Dataset 3: Wieloton 200 Hz, 10 (2000 Hz), 3 składowe # Dataset 3: Wieloton 200 Hz, 10 (2000 Hz), 3 składowe
# Starting x value is 2000 Hz # Starting x value is 2000 Hz
datasets_200Hz.append({ DATASETS_200_HZ.append(
'name': '200 Hz, 10 (2000 Hz), 3 składowe', {
'x': [2000 + 0, 2000 + 50, 2000 + 100, 2000 + 150], "name": "200 Hz, 10 (2000 Hz), 3 składowe",
'y': [225, 215, 220, 265] "x": [2000 + 0, 2000 + 50, 2000 + 100, 2000 + 150],
}) "y": [225, 215, 220, 265],
}
)
# Dataset 4: Wieloton 200 Hz, 10 (2000 Hz), 7 składowych # Dataset 4: Wieloton 200 Hz, 10 (2000 Hz), 7 składowych
datasets_200Hz.append({ DATASETS_200_HZ.append(
'name': '200 Hz, 10 (2000 Hz), 7 składowych', {
'x': [2000 + 0, 2000 + 50, 2000 + 100, 2000 + 150], "name": "200 Hz, 10 (2000 Hz), 7 składowych",
'y': [190, 205, 205, 210] "x": [2000 + 0, 2000 + 50, 2000 + 100, 2000 + 150],
}) "y": [190, 205, 205, 210],
}
)
# Dataset 4: Wieloton 200 Hz, 10 (2000 Hz), 7 składowych # Dataset 4: Wieloton 200 Hz, 10 (2000 Hz), 7 składowych
datasets_200Hz.append({ DATASETS_200_HZ.append(
'name': 'Shouten - 7', {
'x': [1400 + 0, 1400 + 50, 1400 + 100, 1400 + 150], "name": "Shouten - 7",
'y': [200, 207.5, 214, 222] "x": [1400 + 0, 1400 + 50, 1400 + 100, 1400 + 150],
}) "y": [200, 207.5, 214, 222],
}
)
# Dataset 4: Wieloton 200 Hz, 10 (2000 Hz), 7 składowych # Dataset 4: Wieloton 200 Hz, 10 (2000 Hz), 7 składowych
datasets_200Hz.append({ DATASETS_200_HZ.append(
'name': 'Shouten - 10', {
'x': [2000 + 0, 2000 + 50, 2000 + 100, 2000 + 150], "name": "Shouten - 10",
'y': [200, 205, 210, 215] "x": [2000 + 0, 2000 + 50, 2000 + 100, 2000 + 150],
}) "y": [200, 205, 210, 215],
}
)
# Dataset 5: Wieloton 400 Hz, 7 (2800 Hz), 3 składowe # Dataset 5: Wieloton 400 Hz, 7 (2800 Hz), 3 składowe
# Starting x value is 2800 Hz # Starting x value is 2800 Hz
datasets_400Hz.append({ DATASETS_400_HZ.append(
'name': '400 Hz, 7 (2800 Hz), 3 składowe', {
'x': [2800 + 0, 2800 + 100, 2800 + 200, 2800 + 300], "name": "400 Hz, 7 (2800 Hz), 3 składowe",
'y': [405, 410, 410, 405] "x": [2800 + 0, 2800 + 100, 2800 + 200, 2800 + 300],
}) "y": [405, 410, 410, 405],
}
)
# Dataset 6: Wieloton 400 Hz, 7 (2800 Hz), 7 składowych # Dataset 6: Wieloton 400 Hz, 7 (2800 Hz), 7 składowych
datasets_400Hz.append({ DATASETS_400_HZ.append(
'name': '400 Hz, 7 (2800 Hz), 7 składowych', {
'x': [2800 + 0, 2800 + 100, 2800 + 200, 2800 + 300], "name": "400 Hz, 7 (2800 Hz), 7 składowych",
'y': [410, 420, 420, 425] "x": [2800 + 0, 2800 + 100, 2800 + 200, 2800 + 300],
}) "y": [410, 420, 420, 425],
}
)
# Dataset 7: Wieloton 400 Hz, 10 (4000 Hz), 3 składowe # Dataset 7: Wieloton 400 Hz, 10 (4000 Hz), 3 składowe
# Starting x value is 4000 Hz # Starting x value is 4000 Hz
datasets_400Hz.append({ DATASETS_400_HZ.append(
'name': '400 Hz, 10 (4000 Hz), 3 składowe', {
'x': [4000 + 0, 4000 + 100, 4000 + 200, 4000 + 300], "name": "400 Hz, 10 (4000 Hz), 3 składowe",
'y': [407, 419, 425, 425] "x": [4000 + 0, 4000 + 100, 4000 + 200, 4000 + 300],
}) "y": [407, 419, 425, 425],
}
)
# Dataset 8: Wieloton 400 Hz, 10 (4000 Hz), 7 składowych # Dataset 8: Wieloton 400 Hz, 10 (4000 Hz), 7 składowych
# Note: The last two 'Wynik [Hz]' values seem inconsistent. # Note: The last two 'Wynik [Hz]' values seem inconsistent.
# They are much lower than expected based on prior values. # They are much lower than expected based on prior values.
# Assuming there might be typos, and they should be 2200 and 2400 # Assuming there might be typos, and they should be 2200 and 2400
# instead of 220 and 240. # instead of 220 and 240.
datasets_400Hz.append({ DATASETS_400_HZ.append(
'name': '400 Hz, 10 (4000 Hz), 7 składowych', {
'x': [4000 + 0, 4000 + 100, 4000 + 200, 4000 + 300], "name": "400 Hz, 10 (4000 Hz), 7 składowych",
'y': [405, 410, 415, 415] # Original data "x": [4000 + 0, 4000 + 100, 4000 + 200, 4000 + 300],
}) "y": [405, 410, 415, 415], # Original data
}
)
datasets_400Hz.append({ DATASETS_400_HZ.append(
'name': 'Shouten - 7', {
'x': [2800 + 0, 2800 + 100, 2800 + 200, 2800 + 300], "name": "Shouten - 7",
'y': [400, 414, 428, 443] "x": [2800 + 0, 2800 + 100, 2800 + 200, 2800 + 300],
}) "y": [400, 414, 428, 443],
}
)
datasets_400Hz.append({ DATASETS_400_HZ.append(
'name': 'Shouten - 10', {
'x': [4000 + 0, 4000 + 100, 4000 + 200, 4000 + 300], "name": "Shouten - 10",
'y': [400, 410, 420, 430] # Original data "x": [4000 + 0, 4000 + 100, 4000 + 200, 4000 + 300],
}) "y": [400, 410, 420, 430], # Original data
}
)
# Plot the data for 200 Hz # Plot the data for 200 Hz
plt.figure(figsize=(12, 8)) plt.figure(figsize=(12, 8))
for dataset in datasets_200Hz: for dataset in DATASETS_200_HZ:
if 'Shouten' in dataset['name']: if "Shouten" in dataset["name"]:
plt.plot(dataset['x'], dataset['y'], marker='s', color='red', label=dataset['name']) # Shouten datasets plt.plot(
dataset["x"], dataset["y"], marker="s", color="red", label=dataset["name"]
) # Shouten datasets
else: else:
plt.plot(dataset['x'], dataset['y'], marker='o', label=dataset['name']) # Other datasets plt.plot(
plt.xlabel('Frequency [Hz]') dataset["x"], dataset["y"], marker="o", label=dataset["name"]
plt.ylabel('Result [Hz]') ) # Other datasets
plt.title('Combined Graph of Results for 200 Hz') plt.xlabel("Frequency [Hz]")
plt.ylabel("Result [Hz]")
plt.title("Combined Graph of Results for 200 Hz")
plt.legend() plt.legend()
plt.grid(True) plt.grid(visible=True)
plt.savefig('results_200Hz.png') # Save the plot plt.savefig("results_200Hz.png") # Save the plot
plt.show() plt.show()
# Plot the data for 400 Hz # Plot the data for 400 Hz
plt.figure(figsize=(12, 8)) plt.figure(figsize=(12, 8))
for dataset in datasets_400Hz: for dataset in DATASETS_400_HZ:
if 'Shouten' in dataset['name']: if "Shouten" in dataset["name"]:
plt.plot(dataset['x'], dataset['y'], marker='s', color='red', label=dataset['name']) # Shouten datasets plt.plot(
dataset["x"], dataset["y"], marker="s", color="red", label=dataset["name"]
) # Shouten datasets
else: else:
plt.plot(dataset['x'], dataset['y'], marker='o', label=dataset['name']) # Other datasets plt.plot(
plt.xlabel('Frequency [Hz]') dataset["x"], dataset["y"], marker="o", label=dataset["name"]
plt.ylabel('Result [Hz]') ) # Other datasets
plt.title('Combined Graph of Results for 400 Hz') plt.xlabel("Frequency [Hz]")
plt.ylabel("Result [Hz]")
plt.title("Combined Graph of Results for 400 Hz")
plt.legend() plt.legend()
plt.grid(True) plt.grid(visible=True)
plt.savefig('results_400Hz.png') # Save the plot plt.savefig("results_400Hz.png") # Save the plot
plt.show() plt.show()

64
NotProgramming/SPD/spd3/main.py Normal file → Executable file
View File

@ -1,3 +1,6 @@
#!/usr/bin/env python3
"""Plot the SPD lab-3 measurement series."""
import matplotlib.pyplot as plt import matplotlib.pyplot as plt
# Values 1 # Values 1
@ -16,20 +19,22 @@ y3 = [22.06, 15.48, 18.74, 17.71]
plt.figure(figsize=(10, 6)) plt.figure(figsize=(10, 6))
# Plot each line with different styles # Plot each line with different styles
plt.plot(x1, y1, marker='o', linestyle='-', color='blue', label='symetrycznie 500 Hz') plt.plot(x1, y1, marker="o", linestyle="-", color="blue", label="symetrycznie 500 Hz")
plt.plot(x2, y2, marker='s', linestyle='--', color='green', label='asymetrycznie 550 Hz') plt.plot(
plt.plot(x3, y3, marker='^', linestyle='-.', color='red', label='asymetrycznie 450 Hz') x2, y2, marker="s", linestyle="--", color="green", label="asymetrycznie 550 Hz"
)
plt.plot(x3, y3, marker="^", linestyle="-.", color="red", label="asymetrycznie 450 Hz")
# Add labels and title # Add labels and title
plt.xlabel('parametr g', fontsize=40) plt.xlabel("parametr g", fontsize=40)
plt.ylabel('Prod detekcji syngalu [dB]', fontsize=40) plt.ylabel("Prod detekcji syngalu [dB]", fontsize=40)
plt.title('500 Hz', fontsize=48) plt.title("500 Hz", fontsize=48)
# Add a legend # Add a legend
plt.legend(fontsize=32) plt.legend(fontsize=32)
# Optional: Add gridlines # Optional: Add gridlines
plt.grid(True) plt.grid(visible=True)
plt.xticks(fontsize=32) plt.xticks(fontsize=32)
plt.yticks(fontsize=32) plt.yticks(fontsize=32)
@ -52,20 +57,22 @@ y3 = [13.0, 13.06, 7.8, 1.7]
plt.figure(figsize=(10, 6)) plt.figure(figsize=(10, 6))
# Plot each line with different styles # Plot each line with different styles
plt.plot(x1, y1, marker='o', linestyle='-', color='blue', label='symetrycznie 1500 Hz') plt.plot(x1, y1, marker="o", linestyle="-", color="blue", label="symetrycznie 1500 Hz")
plt.plot(x2, y2, marker='s', linestyle='--', color='green', label='asymetrycznie 1650 Hz') plt.plot(
plt.plot(x3, y3, marker='^', linestyle='-.', color='red', label='asymetrycznie 1350 Hz') x2, y2, marker="s", linestyle="--", color="green", label="asymetrycznie 1650 Hz"
)
plt.plot(x3, y3, marker="^", linestyle="-.", color="red", label="asymetrycznie 1350 Hz")
# Add labels and title # Add labels and title
plt.xlabel('parametr g', fontsize=40) plt.xlabel("parametr g", fontsize=40)
plt.ylabel('Prod detekcji syngalu [dB]', fontsize=40) plt.ylabel("Prod detekcji syngalu [dB]", fontsize=40)
plt.title('1500 Hz', fontsize=48) plt.title("1500 Hz", fontsize=48)
# Add a legend # Add a legend
plt.legend(fontsize=32) plt.legend(fontsize=32)
# Optional: Add gridlines # Optional: Add gridlines
plt.grid(True) plt.grid(visible=True)
plt.xticks(fontsize=32) plt.xticks(fontsize=32)
plt.yticks(fontsize=32) plt.yticks(fontsize=32)
@ -84,19 +91,21 @@ y2 = [33.06, 35.28, 23.33, 17.14, 14.33, 10.01, 4.43]
plt.figure(figsize=(10, 6)) plt.figure(figsize=(10, 6))
# Plot each line with different styles # Plot each line with different styles
plt.plot(x1, y1, marker='o', linestyle='-', color='blue', label='symetrycznie 500 Hz') plt.plot(x1, y1, marker="o", linestyle="-", color="blue", label="symetrycznie 500 Hz")
plt.plot(x2, y2, marker='s', linestyle='--', color='green', label='symetrycznie 1500 Hz') plt.plot(
x2, y2, marker="s", linestyle="--", color="green", label="symetrycznie 1500 Hz"
)
# Add labels and title # Add labels and title
plt.xlabel('parametr g', fontsize=40) plt.xlabel("parametr g", fontsize=40)
plt.ylabel('Prod detekcji syngalu [dB]', fontsize=40) plt.ylabel("Prod detekcji syngalu [dB]", fontsize=40)
plt.title('500 Hz vs 1500 Hz', fontsize=48) plt.title("500 Hz vs 1500 Hz", fontsize=48)
# Add a legend # Add a legend
plt.legend(fontsize=32) plt.legend(fontsize=32)
# Optional: Add gridlines # Optional: Add gridlines
plt.grid(True) plt.grid(visible=True)
plt.xticks(fontsize=32) plt.xticks(fontsize=32)
plt.yticks(fontsize=32) plt.yticks(fontsize=32)
@ -115,22 +124,23 @@ y2 = [81.59, 60.08, 68.48, 58.77, 53.6, 55.46]
plt.figure(figsize=(10, 6)) plt.figure(figsize=(10, 6))
# Plot each line with different styles # Plot each line with different styles
plt.plot(x1, y1, marker='o', linestyle='-', color='blue', label='pasmowy 500 Hz') plt.plot(x1, y1, marker="o", linestyle="-", color="blue", label="pasmowy 500 Hz")
plt.plot(x2, y2, marker='s', linestyle='--', color='green', label='pasmowo-zaporowy 500 Hz') plt.plot(
x2, y2, marker="s", linestyle="--", color="green", label="pasmowo-zaporowy 500 Hz"
)
# Add labels and title # Add labels and title
plt.xlabel('parametr g', fontsize=40) plt.xlabel("parametr g", fontsize=40)
plt.ylabel('Prod detekcji syngalu [dB]', fontsize=40) plt.ylabel("Prod detekcji syngalu [dB]", fontsize=40)
plt.title('Porownanie szum pasmowy vs szum pasmowo zaporowy dla 500 Hz', fontsize=48) plt.title("Porownanie szum pasmowy vs szum pasmowo zaporowy dla 500 Hz", fontsize=48)
# Add a legend # Add a legend
plt.legend(fontsize=32) plt.legend(fontsize=32)
# Optional: Add gridlines # Optional: Add gridlines
plt.grid(True) plt.grid(visible=True)
plt.xticks(fontsize=32) plt.xticks(fontsize=32)
plt.yticks(fontsize=32) plt.yticks(fontsize=32)
# Display the plot # Display the plot
plt.show() plt.show()

21
NotProgramming/SPD/spd_5/main.py Normal file → Executable file
View File

@ -1,3 +1,6 @@
#!/usr/bin/env python3
"""Plot the SPD lab-5 measurement series."""
import matplotlib.pyplot as plt import matplotlib.pyplot as plt
import numpy as np import numpy as np
@ -23,18 +26,20 @@ y_pure_tone_line = 9 * log_delta_t_line - 19
plt.figure(figsize=(10, 6)) plt.figure(figsize=(10, 6))
# Plot the experimental data # Plot the experimental data
plt.semilogx(delta_t, ton_60_db_minus, 'o', label='Ton eksperymentalny 60 dB') plt.semilogx(delta_t, ton_60_db_minus, "o", label="Ton eksperymentalny 60 dB")
plt.semilogx(delta_t, szum_20_db_per_hz_minus, 's', label='Szum eksperymentalny 20 dB/Hz') plt.semilogx(
delta_t, szum_20_db_per_hz_minus, "s", label="Szum eksperymentalny 20 dB/Hz"
)
# Plot the theoretical lines # Plot the theoretical lines
plt.semilogx(delta_t_line, y_pure_tone_line, label='Ton - krzywa teoretyczna') plt.semilogx(delta_t_line, y_pure_tone_line, label="Ton - krzywa teoretyczna")
plt.semilogx(delta_t_line, y_broadband_line, label='Szum - krzywa teoereyczna') plt.semilogx(delta_t_line, y_broadband_line, label="Szum - krzywa teoereyczna")
plt.xlabel('Czas trwania sygnału [ms]', fontsize=40) plt.xlabel("Czas trwania sygnału [ms]", fontsize=40)
plt.ylabel('Poziom względny głośności [dB]', fontsize=40) plt.ylabel("Poziom względny głośności [dB]", fontsize=40)
plt.title('Głośnośc vs czas', fontsize=48) plt.title("Głośnośc vs czas", fontsize=48)
plt.legend(fontsize=32) plt.legend(fontsize=32)
plt.grid(True, which="both", ls="--") plt.grid(visible=True, which="both", ls="--")
plt.xticks(fontsize=32) plt.xticks(fontsize=32)
plt.yticks(fontsize=32) plt.yticks(fontsize=32)

27
NotProgramming/SPD/spd_5/main2.py Normal file → Executable file
View File

@ -1,24 +1,27 @@
#!/usr/bin/env python3
"""Plot the SPD lab-5 detection thresholds at 600, 2200 and 4400 Hz."""
import matplotlib.pyplot as plt import matplotlib.pyplot as plt
import numpy as np import numpy as np
# Data (decreasing values) # Data (decreasing values)
bandwidth = np.array([40, 100, 160, 300, 600, 1000]) bandwidth = np.array([40, 100, 160, 300, 600, 1000])
detection_threshold_600Hz = np.array([50, 30, 20, 12, 10, 8]) DETECTION_THRESHOLD_600_HZ = np.array([50, 30, 20, 12, 10, 8])
detection_threshold_2200Hz = np.array([40, 25, 18, 10, 8, 7]) DETECTION_THRESHOLD_2200_HZ = np.array([40, 25, 18, 10, 8, 7])
detection_threshold_4400Hz = np.array([35, 22, 15, 9, 7, 6]) DETECTION_THRESHOLD_4400_HZ = np.array([35, 22, 15, 9, 7, 6])
# Plot # Plot
plt.figure(figsize=(10, 6)) plt.figure(figsize=(10, 6))
plt.xscale('log') plt.xscale("log")
plt.scatter(bandwidth, detection_threshold_600Hz, label='600 Hz', marker='o') plt.scatter(bandwidth, DETECTION_THRESHOLD_600_HZ, label="600 Hz", marker="o")
plt.scatter(bandwidth, detection_threshold_2200Hz, label='2200 Hz', marker='^') plt.scatter(bandwidth, DETECTION_THRESHOLD_2200_HZ, label="2200 Hz", marker="^")
plt.scatter(bandwidth, detection_threshold_4400Hz, label='4400 Hz', marker='s') plt.scatter(bandwidth, DETECTION_THRESHOLD_4400_HZ, label="4400 Hz", marker="s")
plt.xlabel('Szerokość pasma [Hz]') plt.xlabel("Szerokość pasma [Hz]")
plt.ylabel('Próg detekcji interwału ciszy [ms]') plt.ylabel("Próg detekcji interwału ciszy [ms]")
plt.title('Górna częstotliwość odcięcia (GCO)') plt.title("Górna częstotliwość odcięcia (GCO)")
plt.legend(title='GCO') plt.legend(title="GCO")
plt.grid(True) plt.grid(visible=True)
plt.show() plt.show()

93
Programming/EARIN/lab1/main.py
View File

@ -1,5 +1,4 @@
""" """Write horizontal_line program that solves horizontal_line maze
Write horizontal_line program that solves horizontal_line maze
using greedy best-first search algorithm. using greedy best-first search algorithm.
The maze is horizontal_line 2D grid The maze is horizontal_line 2D grid
with empty space, walls, horizontal_line start, and an end position. with empty space, walls, horizontal_line start, and an end position.
@ -24,14 +23,14 @@ Does not work if no path (Should print out NO PATH FOUND)
""" """
import heapq import heapq
import os
import sys import sys
import time import time
import os from random import random, randrange, shuffle
from random import shuffle, randrange, random
class MazeSolver: class MazeSolver:
"""Maze Solver""" """Maze Solver."""
# self corresponds to "this" in js, it refers to object of MazeSolver class # self corresponds to "this" in js, it refers to object of MazeSolver class
@ -45,7 +44,7 @@ class MazeSolver:
# Start/End character # Start/End character
def find_start_and_end(self): def find_start_and_end(self):
"""Finds start and end points in the maze""" """Finds start and end points in the maze."""
start = end = None start = end = None
for row_i, row in enumerate(self.maze): for row_i, row in enumerate(self.maze):
@ -66,7 +65,7 @@ class MazeSolver:
# If it is not horizontal_line "wall" (#) add its position to list of neighbors # If it is not horizontal_line "wall" (#) add its position to list of neighbors
def get_neighbors(self, position): def get_neighbors(self, position):
"""Finds point'maze_data neighbors""" """Finds point'maze_data neighbors."""
row, col = position row, col = position
neighbors = [] neighbors = []
if row > 0 and self.maze[row - 1][col] != "#": if row > 0 and self.maze[row - 1][col] != "#":
@ -82,7 +81,7 @@ class MazeSolver:
# find path through maze # find path through maze
def solve_loop(self, queue, visited): def solve_loop(self, queue, visited):
""" Goes through maze and finds the path """ """Goes through maze and finds the path."""
heuristic_total_time = 0 heuristic_total_time = 0
heuristics_called = 0 heuristics_called = 0
while queue: while queue:
@ -100,20 +99,17 @@ class MazeSolver:
for neighbor in self.get_neighbors(current): for neighbor in self.get_neighbors(current):
if neighbor not in visited: if neighbor not in visited:
new_path = path + [neighbor] new_path = path + [neighbor]
heuristic, heuristic_time = self.heuristic_euclidean( heuristic, heuristic_time = self.heuristic_euclidean(neighbor)
neighbor)
heuristic_total_time += heuristic_time heuristic_total_time += heuristic_time
heuristics_called += 1 heuristics_called += 1
heapq.heappush( heapq.heappush(queue, (heuristic, neighbor, new_path))
queue, (heuristic, neighbor, new_path)
)
if not self.test: if not self.test:
print_maze(self.maze, path, visited) print_maze(self.maze, path, visited)
print() print()
return path, visited, heuristic_total_time, heuristics_called return path, visited, heuristic_total_time, heuristics_called
def solve(self): def solve(self):
"""Solves the maze""" """Solves the maze."""
queue = [] queue = []
# set means that values inside can not repeat # set means that values inside can not repeat
visited = set() visited = set()
@ -122,9 +118,7 @@ class MazeSolver:
# we use heapq so the element with lowest heuristic value will always # we use heapq so the element with lowest heuristic value will always
# be at the top of heap # be at the top of heap
heuristic = self.heuristic_euclidean(self.start) heuristic = self.heuristic_euclidean(self.start)
heapq.heappush( heapq.heappush(queue, (heuristic, self.start, [self.start]))
queue, (heuristic, self.start, [self.start])
)
# Go through queue until it'maze_data empty # Go through queue until it'maze_data empty
# Find neighbor (which is not wall) closest to the # Find neighbor (which is not wall) closest to the
@ -137,10 +131,9 @@ class MazeSolver:
# This heuristic returns the Manhattan distance between the given position # This heuristic returns the Manhattan distance between the given position
# and the maze'maze_data end # and the maze'maze_data end
def heuristic_manhattan(self, position): def heuristic_manhattan(self, position):
"""Heuristic function that uses Manhattan distance""" """Heuristic function that uses Manhattan distance."""
start_time = time.perf_counter() start_time = time.perf_counter()
heuristic = abs(position[0] - self.end[0]) + \ heuristic = abs(position[0] - self.end[0]) + abs(position[1] - self.end[1])
abs(position[1] - self.end[1])
end_time = time.perf_counter() end_time = time.perf_counter()
heuristic_time = end_time - start_time heuristic_time = end_time - start_time
return heuristic, heuristic_time return heuristic, heuristic_time
@ -148,18 +141,17 @@ class MazeSolver:
# This heuristic returns the Euclidean distance between the given position # This heuristic returns the Euclidean distance between the given position
# and the maze'maze_data end # and the maze'maze_data end
def heuristic_euclidean(self, position): def heuristic_euclidean(self, position):
"""Heuristic function that uses Euclidean distance""" """Heuristic function that uses Euclidean distance."""
start_time = time.perf_counter() start_time = time.perf_counter()
heuristic = ( heuristic = (
abs(position[0] - self.end[0]) ** 2 + abs(position[0] - self.end[0]) ** 2 + abs(position[1] - self.end[1]) ** 2
abs(position[1] - self.end[1]) ** 2
) ** 0.5 ) ** 0.5
end_time = time.perf_counter() end_time = time.perf_counter()
heuristic_time = end_time - start_time heuristic_time = end_time - start_time
return heuristic, heuristic_time return heuristic, heuristic_time
def heuristic_random(self, position): def heuristic_random(self, position):
"""Heuristic function that just returns random value between 0 and 1""" """Heuristic function that just returns random value between 0 and 1."""
start_time = time.perf_counter() start_time = time.perf_counter()
heuristic = random() heuristic = random()
end_time = time.perf_counter() end_time = time.perf_counter()
@ -169,9 +161,9 @@ class MazeSolver:
# Open and load text file to array # Open and load text file to array
def load_maze(maze_file_name): def load_maze(maze_file_name):
"""Loads horizontal_line maze from the specified file""" """Loads horizontal_line maze from the specified file."""
# Open for reading only and save to fileContents # Open for reading only and save to fileContents
with open(maze_file_name, "r", encoding="utf8") as file_contents: with open(maze_file_name, encoding="utf8") as file_contents:
# strip() removes extra white spaces from the beginning and the end of # strip() removes extra white spaces from the beginning and the end of
# horizontal_line string # horizontal_line string
# list() changes string to array of chars # list() changes string to array of chars
@ -184,7 +176,7 @@ def load_maze(maze_file_name):
def print_maze(maze, path=None, visited=None): def print_maze(maze, path=None, visited=None):
"""Prints the maze""" """Prints the maze."""
if path is None: if path is None:
path = [] path = []
if visited is None: if visited is None:
@ -201,7 +193,7 @@ def print_maze(maze, path=None, visited=None):
def create_maze_folder(solved): def create_maze_folder(solved):
""" Creates folder for generated or solved mazes""" """Creates folder for generated or solved mazes."""
if solved: if solved:
folder_name = "solvedMazes" folder_name = "solvedMazes"
else: else:
@ -211,14 +203,18 @@ def create_maze_folder(solved):
return folder_name return folder_name
def save_maze(maze, solved=True, path=None, visited=None, saved_file="Maze", iteration=0): def save_maze(
"""Saves maze from array to txt file""" maze, solved=True, path=None, visited=None, saved_file="Maze", iteration=0
):
"""Saves maze from array to txt file."""
folder_name = create_maze_folder(solved) folder_name = create_maze_folder(solved)
if path is None: if path is None:
path = [] path = []
if visited is None: if visited is None:
visited = [] visited = []
with open(f"{folder_name}/{iteration}{os.path.basename(saved_file)}", "w", encoding="utf8") as maze_file: with open(
f"{folder_name}/{iteration}{os.path.basename(saved_file)}", "w", encoding="utf8"
) as maze_file:
for row_i, row in enumerate(maze): for row_i, row in enumerate(maze):
for col_i, cell in enumerate(row): for col_i, cell in enumerate(row):
if (row_i, col_i) in path and cell == " ": if (row_i, col_i) in path and cell == " ":
@ -234,7 +230,7 @@ def save_maze(maze, solved=True, path=None, visited=None, saved_file="Maze", ite
def fill_generated_maze(hor, ver, width): def fill_generated_maze(hor, ver, width):
""" Fills generated maze array from horizontal and vertical lines """ """Fills generated maze array from horizontal and vertical lines."""
maze_data = "" maze_data = ""
for horizontal_line, vertical_line in zip(hor, ver): for horizontal_line, vertical_line in zip(hor, ver):
maze_data += "".join(horizontal_line + ["\n"] + vertical_line + ["\n"]) maze_data += "".join(horizontal_line + ["\n"] + vertical_line + ["\n"])
@ -246,7 +242,7 @@ def fill_generated_maze(hor, ver, width):
def make_maze(width=16, height=8): def make_maze(width=16, height=8):
""" generate maze with given width and height """ """Generate maze with given width and height."""
vis = [[0] * width + [1] for _ in range(height)] + [[1] * (width + 1)] vis = [[0] * width + [1] for _ in range(height)] + [[1] * (width + 1)]
ver = [["# "] * width + ["#"] for _ in range(height)] + [[]] ver = [["# "] * width + ["#"] for _ in range(height)] + [[]]
hor = [["###"] * width + ["#"] for _ in range(height + 1)] hor = [["###"] * width + ["#"] for _ in range(height + 1)]
@ -254,20 +250,20 @@ def make_maze(width=16, height=8):
def walk(x_coordinate, y_coordinate): def walk(x_coordinate, y_coordinate):
vis[y_coordinate][x_coordinate] = 1 vis[y_coordinate][x_coordinate] = 1
neighbors = [(x_coordinate - 1, y_coordinate), neighbors = [
(x_coordinate, y_coordinate + 1), (x_coordinate - 1, y_coordinate),
(x_coordinate + 1, y_coordinate), (x_coordinate, y_coordinate + 1),
(x_coordinate, y_coordinate - 1)] (x_coordinate + 1, y_coordinate),
(x_coordinate, y_coordinate - 1),
]
shuffle(neighbors) shuffle(neighbors)
for x_coordinate_neighbor, y_coordinate_neighbor in neighbors: for x_coordinate_neighbor, y_coordinate_neighbor in neighbors:
if vis[y_coordinate_neighbor][x_coordinate_neighbor]: if vis[y_coordinate_neighbor][x_coordinate_neighbor]:
continue continue
if x_coordinate_neighbor == x_coordinate: if x_coordinate_neighbor == x_coordinate:
hor[max(y_coordinate, y_coordinate_neighbor) hor[max(y_coordinate, y_coordinate_neighbor)][x_coordinate] = "# "
][x_coordinate] = "# "
if y_coordinate_neighbor == y_coordinate: if y_coordinate_neighbor == y_coordinate:
ver[y_coordinate][max( ver[y_coordinate][max(x_coordinate, x_coordinate_neighbor)] = " "
x_coordinate, x_coordinate_neighbor)] = " "
walk(x_coordinate_neighbor, y_coordinate_neighbor) walk(x_coordinate_neighbor, y_coordinate_neighbor)
walk(randrange(width), randrange(height)) walk(randrange(width), randrange(height))
@ -276,7 +272,7 @@ def make_maze(width=16, height=8):
def print_help(): def print_help():
"""prints help""" """Prints help."""
print( print(
"""python main.py - run the script against default maze file """python main.py - run the script against default maze file
(any file named maze.txt in the code directory) (any file named maze.txt in the code directory)
@ -296,7 +292,7 @@ Number parameter and puts it in the generatedMazes folder"""
def test_mode(): def test_mode():
""" Loads and solves multiple mazes in order to compare heuristics """ """Loads and solves multiple mazes in order to compare heuristics."""
create_maze_folder(False) create_maze_folder(False)
sum_of_paths = 0 sum_of_paths = 0
files_amount = 0 files_amount = 0
@ -311,7 +307,9 @@ def test_mode():
solver_test = MazeSolver(loaded_maze, TEST_MODE) solver_test = MazeSolver(loaded_maze, TEST_MODE)
# Find path using MazeSolver solve method # Find path using MazeSolver solve method
start_time = time.perf_counter() start_time = time.perf_counter()
solved_path, visited, heuristic_total_time, heuristics_called = solver_test.solve() solved_path, visited, heuristic_total_time, heuristics_called = (
solver_test.solve()
)
heuristic_total_total_time += heuristic_total_time heuristic_total_total_time += heuristic_total_time
all_heuristic_called += heuristics_called all_heuristic_called += heuristics_called
end_time = time.perf_counter() end_time = time.perf_counter()
@ -335,7 +333,7 @@ def test_mode():
def default(): def default():
""" Runs default operation - reads, solves and prints single maze from file """ """Runs default operation - reads, solves and prints single maze from file."""
# Open and load text file to array # Open and load text file to array
loaded_maze = load_maze(FILE_NAME) loaded_maze = load_maze(FILE_NAME)
# Initialize MazeSolver object with maze as parameter # Initialize MazeSolver object with maze as parameter
@ -365,13 +363,12 @@ if __name__ == "__main__":
FOLDER_NAME = sys.argv[2] FOLDER_NAME = sys.argv[2]
test_mode() test_mode()
sys.exit() sys.exit()
if sys.argv[1] == '-g' or sys.argv[1] == '--generate': if sys.argv[1] == "-g" or sys.argv[1] == "--generate":
if len(sys.argv) > 2: if len(sys.argv) > 2:
GENERATE_AMOUNT = int(sys.argv[2]) GENERATE_AMOUNT = int(sys.argv[2])
for n in range(GENERATE_AMOUNT): for n in range(GENERATE_AMOUNT):
GENERATED_MAZE = make_maze() GENERATED_MAZE = make_maze()
save_maze(GENERATED_MAZE, False, None, save_maze(GENERATED_MAZE, False, None, None, f"generated{n}.txt")
None, f'generated{n}.txt')
sys.exit() sys.exit()
FILE_NAME = sys.argv[1] FILE_NAME = sys.argv[1]
default() default()

104
Programming/EARIN/lab2/main.py
View File

@ -1,14 +1,13 @@
""" """Program that plays draughts (checkers) with user on 8x8 board using min-max with alpha-beta pruning."""
Program that plays draughts (checkers) with user on 8x8 board using min-max with alpha-beta pruning
"""
import re
import copy import copy
import math import math
import re
import sys import sys
class Game: class Game:
"""Game""" """Game."""
def __init__(self, size): def __init__(self, size):
self.board_size = size self.board_size = size
@ -16,20 +15,19 @@ class Game:
self.black_positions = self.initialize_black() self.black_positions = self.initialize_black()
def initialize_white(self): def initialize_white(self):
"""Initialize white pieces""" """Initialize white pieces."""
white_positions = [] white_positions = []
for y_coordinate in range(math.floor((self.board_size - 2) / 2)): for y_coordinate in range(math.floor((self.board_size - 2) / 2)):
for x_coordinate in range(self.board_size): for x_coordinate in range(self.board_size):
if y_coordinate % 2 == 0: if y_coordinate % 2 == 0:
if x_coordinate % 2 == 1: if x_coordinate % 2 == 1:
white_positions.append((x_coordinate, y_coordinate, False)) white_positions.append((x_coordinate, y_coordinate, False))
else: elif x_coordinate % 2 == 0:
if x_coordinate % 2 == 0: white_positions.append((x_coordinate, y_coordinate, False))
white_positions.append((x_coordinate, y_coordinate, False))
return white_positions return white_positions
def initialize_black(self): def initialize_black(self):
"""Initialize black pieces""" """Initialize black pieces."""
black_positions = [] black_positions = []
for y_coordinate in range( for y_coordinate in range(
self.board_size - math.floor((self.board_size - 2) / 2), self.board_size self.board_size - math.floor((self.board_size - 2) / 2), self.board_size
@ -38,13 +36,12 @@ class Game:
if y_coordinate % 2 == 0: if y_coordinate % 2 == 0:
if x_coordinate % 2 == 1: if x_coordinate % 2 == 1:
black_positions.append((x_coordinate, y_coordinate, False)) black_positions.append((x_coordinate, y_coordinate, False))
else: elif x_coordinate % 2 == 0:
if x_coordinate % 2 == 0: black_positions.append((x_coordinate, y_coordinate, False))
black_positions.append((x_coordinate, y_coordinate, False))
return black_positions return black_positions
def check_move_out_of_bounds(self, to_): def check_move_out_of_bounds(self, to_):
"""Check if the move destination is out of the bounds of the board""" """Check if the move destination is out of the bounds of the board."""
if to_[0] < 0 or to_[0] > self.board_size - 1: if to_[0] < 0 or to_[0] > self.board_size - 1:
# print(f"Illegal move! Final x coordinate must be between 0 and {self.board_size-1}!") # print(f"Illegal move! Final x coordinate must be between 0 and {self.board_size-1}!")
return True return True
@ -54,24 +51,24 @@ class Game:
return False return False
def check_piece_exists(self, coords, color): def check_piece_exists(self, coords, color):
"""Check if a piece of given color exists at a given spot""" """Check if a piece of given color exists at a given spot."""
if color == "white": if color == "white":
if any( if any(
piece in self.white_positions piece in self.white_positions
for piece in ((*coords, False), (*coords, True)) for piece in ((*coords, False), (*coords, True))
): ):
return True return True
else: elif any(
if any( piece in self.black_positions
piece in self.black_positions for piece in ((*coords, False), (*coords, True))
for piece in ((*coords, False), (*coords, True)) ):
): return True
return True
return False return False
def check_piece_king(self, coords, color): def check_piece_king(self, coords, color):
"""Check if a piece of in a given spot and of a given color is a king. """Check if a piece of in a given spot and of a given color is a king.
Return false if no piece is found""" Return false if no piece is found.
"""
if color == "white": if color == "white":
return (*coords, True) in self.white_positions return (*coords, True) in self.white_positions
if color == "black": if color == "black":
@ -80,7 +77,7 @@ class Game:
# https://stackoverflow.com/a/2191707 # https://stackoverflow.com/a/2191707
def check_move_piece_capable(self, from_, to_, color): def check_move_piece_capable(self, from_, to_, color):
"""Check if the move is exactly one square diagonally""" """Check if the move is exactly one square diagonally."""
if abs(to_[0] - from_[0]) == 1: if abs(to_[0] - from_[0]) == 1:
if self.check_piece_king(from_, color): if self.check_piece_king(from_, color):
return True return True
@ -92,7 +89,8 @@ class Game:
def check_capture(self, from_, to_, color): def check_capture(self, from_, to_, color):
"""Check if a piece was captured for a given move. """Check if a piece was captured for a given move.
Return captured piece coordinates or None""" Return captured piece coordinates or None.
"""
# captures can only happen if the player moved twice-diagonally # captures can only happen if the player moved twice-diagonally
if abs(to_[0] - from_[0]) != 2 or abs(to_[1] - from_[1]) != 2: if abs(to_[0] - from_[0]) != 2 or abs(to_[1] - from_[1]) != 2:
@ -107,7 +105,7 @@ class Game:
return None return None
def check_move_legal(self, from_, to_, color, give_feedback=False): def check_move_legal(self, from_, to_, color, give_feedback=False):
"""Check if a move is legal. Return a boolean or coordinates of captured piece""" """Check if a move is legal. Return a boolean or coordinates of captured piece."""
if self.check_move_out_of_bounds(to_): if self.check_move_out_of_bounds(to_):
if give_feedback: if give_feedback:
print("Illegal move! Final position is out of the bounds of the board") print("Illegal move! Final position is out of the bounds of the board")
@ -135,7 +133,7 @@ class Game:
return capture return capture
def make_move(self, from_, to_, color): def make_move(self, from_, to_, color):
"""Move a piece on the board and remove any captured pieces""" """Move a piece on the board and remove any captured pieces."""
move_legal = self.check_move_legal(from_, to_, color) move_legal = self.check_move_legal(from_, to_, color)
if move_legal is False: if move_legal is False:
return False return False
@ -165,19 +163,19 @@ class Game:
return True return True
def print_board(self, rotate=False): def print_board(self, rotate=False):
"""Print the board in the console""" """Print the board in the console."""
def print_letters(): def print_letters():
"""Print the letters above or under the board""" """Print the letters above or under the board."""
print(" ", end="") print(" ", end="")
for col in range(self.board_size): for col in range(self.board_size):
if rotate: if rotate:
print(f" {chr(ord('a')+self.board_size-1-col)}", end=" ") print(f" {chr(ord('a') + self.board_size - 1 - col)}", end=" ")
else: else:
print(f" {chr(ord('a')+col)}", end=" ") print(f" {chr(ord('a') + col)}", end=" ")
def get_square_code(pos, background): def get_square_code(pos, background):
"""Return the code of a given square on the board""" """Return the code of a given square on the board."""
if (*pos, False) in self.white_positions: if (*pos, False) in self.white_positions:
return "w" return "w"
if (*pos, True) in self.white_positions: if (*pos, True) in self.white_positions:
@ -210,7 +208,7 @@ class Game:
if col == 0: if col == 0:
if line % 4 == 2: if line % 4 == 2:
print(f"{row//4:3d}", end="") print(f"{row // 4:3d}", end="")
else: else:
print(" ", end="") print(" ", end="")
@ -218,13 +216,13 @@ class Game:
case 0: case 0:
print("+---", end="") print("+---", end="")
case 1 | 3: case 1 | 3:
print(f"|{3*background}", end="") print(f"|{3 * background}", end="")
case 2: case 2:
print(f"|{background}{checker}{background}", end="") print(f"|{background}{checker}{background}", end="")
if line % 4 == 0: if line % 4 == 0:
print("+") print("+")
else: else:
print(f"|{row//4}" if line % 4 == 2 else "|") print(f"|{row // 4}" if line % 4 == 2 else "|")
line += 1 line += 1
print(" ", end="") print(" ", end="")
for col in range(self.board_size): for col in range(self.board_size):
@ -235,7 +233,7 @@ class Game:
# Ran first in the code # Ran first in the code
def get_possible_moves_capture(self, from_, color): def get_possible_moves_capture(self, from_, color):
"""Return all possible captures for a piece""" """Return all possible captures for a piece."""
# all capturing moves: # all capturing moves:
legal_moves = [] legal_moves = []
move_down_left_two = (from_[0] + 2, from_[1] - 2) move_down_left_two = (from_[0] + 2, from_[1] - 2)
@ -253,7 +251,7 @@ class Game:
return legal_moves return legal_moves
def get_possible_moves_non_capture(self, from_, color): def get_possible_moves_non_capture(self, from_, color):
"""Return all possible moves that are not captures for a piece""" """Return all possible moves that are not captures for a piece."""
# all non-capturing moves # all non-capturing moves
legal_moves = [] legal_moves = []
move_down_left_one = (from_[0] + 1, from_[1] - 1) move_down_left_one = (from_[0] + 1, from_[1] - 1)
@ -271,7 +269,7 @@ class Game:
return legal_moves return legal_moves
def get_possible_moves(self, color): def get_possible_moves(self, color):
"""Return all possible moves for a given piece color""" """Return all possible moves for a given piece color."""
legal_moves = [] legal_moves = []
captures = [] captures = []
if color == "white": if color == "white":
@ -298,7 +296,8 @@ class Game:
def alpha_beta(self, depth, alpha_beta, color, current_color=None): def alpha_beta(self, depth, alpha_beta, color, current_color=None):
"""Do alpha beta pruning for given parameters """Do alpha beta pruning for given parameters
and return the best move and its evaluated points""" and return the best move and its evaluated points.
"""
if current_color is None: if current_color is None:
current_color = color current_color = color
@ -353,7 +352,7 @@ class Game:
return None return None
def evaluate(self, color): def evaluate(self, color):
"""Evaluates the state of the board for a given color""" """Evaluates the state of the board for a given color."""
white_score = 0 white_score = 0
black_score = 0 black_score = 0
@ -374,13 +373,13 @@ class Game:
) )
def input_to_coordinates(self, user_input): def input_to_coordinates(self, user_input):
"""Change input from a1 form to tuple form""" """Change input from a1 form to tuple form."""
pos_x = ord(user_input[0]) - ord("a") pos_x = ord(user_input[0]) - ord("a")
pos_y = int(user_input[1::]) pos_y = int(user_input[1::])
return pos_x, pos_y return pos_x, pos_y
def handle_player_move(self, color): def handle_player_move(self, color):
"""Prompt player to move, validate their input and make move""" """Prompt player to move, validate their input and make move."""
has_moved = False has_moved = False
possible_moves = self.get_possible_moves(color)[0] possible_moves = self.get_possible_moves(color)[0]
while not has_moved: while not has_moved:
@ -396,7 +395,7 @@ class Game:
from_coordinates = self.input_to_coordinates(move_from) from_coordinates = self.input_to_coordinates(move_from)
to_coordinates = self.input_to_coordinates(move_to) to_coordinates = self.input_to_coordinates(move_to)
if not (from_coordinates, to_coordinates) in possible_moves: if (from_coordinates, to_coordinates) not in possible_moves:
legal_no_captures = self.check_move_legal( legal_no_captures = self.check_move_legal(
from_coordinates, to_coordinates, color, True from_coordinates, to_coordinates, color, True
) )
@ -408,7 +407,7 @@ class Game:
self.print_board(color == "white") self.print_board(color == "white")
def start_game(self, player_color="black", algorithm_depth=5): def start_game(self, player_color="black", algorithm_depth=5):
"""Start the main loop of the game""" """Start the main loop of the game."""
if player_color not in ("black", "white"): if player_color not in ("black", "white"):
print("Invalid color! Color can be black or white") print("Invalid color! Color can be black or white")
return return
@ -428,8 +427,8 @@ class Game:
game.make_move(*ai_move, ai_color) game.make_move(*ai_move, ai_color)
print( print(
"AI's move: " "AI's move: "
f"{chr(ord('a')+ai_move[0][0])}{ai_move[0][1]} " f"{chr(ord('a') + ai_move[0][0])}{ai_move[0][1]} "
f"{chr(ord('a')+ai_move[1][0])}{ai_move[1][1]}" f"{chr(ord('a') + ai_move[1][0])}{ai_move[1][1]}"
) )
game.print_board(player_color == "white") game.print_board(player_color == "white")
ai_turn = game.get_possible_moves(ai_color)[1] and possible_moves_ai[1] ai_turn = game.get_possible_moves(ai_color)[1] and possible_moves_ai[1]
@ -446,7 +445,7 @@ class Game:
) )
def ai_turn(self, ai_color, algorithm_depth, possible_moves_ai, print_info): def ai_turn(self, ai_color, algorithm_depth, possible_moves_ai, print_info):
""" Calculates ai move and makes it """ """Calculates ai move and makes it."""
if len(possible_moves_ai) == 0: if len(possible_moves_ai) == 0:
if print_info: if print_info:
print(f"Game over, {ai_color} loses") print(f"Game over, {ai_color} loses")
@ -458,14 +457,14 @@ class Game:
if print_info: if print_info:
print( print(
"AI's move: " "AI's move: "
f"{chr(ord('a')+ai_move[0][0])}{ai_move[0][1]} " f"{chr(ord('a') + ai_move[0][0])}{ai_move[0][1]} "
f"{chr(ord('a')+ai_move[1][0])}{ai_move[1][1]}" f"{chr(ord('a') + ai_move[1][0])}{ai_move[1][1]}"
) )
game.print_board(True) game.print_board(True)
return False return False
def auto_game(self, white_depth, black_depth): def auto_game(self, white_depth, black_depth):
"""Auto game mode between two bots""" """Auto game mode between two bots."""
game_turn = 0 game_turn = 0
max_turns = 250 max_turns = 250
while game_turn < max_turns: while game_turn < max_turns:
@ -491,8 +490,9 @@ class Game:
return "" return ""
return "" return ""
def auto_simulation(white_depth, black_depth, iterations): def auto_simulation(white_depth, black_depth, iterations):
"""Runs iterations amount of simulations""" """Runs iterations amount of simulations."""
print( print(
f"""Running {iterations} simulations with f"""Running {iterations} simulations with
white depth = {white_depth},black depth = {black_depth}""" white depth = {white_depth},black depth = {black_depth}"""
@ -522,7 +522,7 @@ def auto_simulation(white_depth, black_depth, iterations):
def print_help(): def print_help():
"""prints help""" """Prints help."""
print( print(
"""python main.py [algorithm_depth] - play the game against the bot as black, """python main.py [algorithm_depth] - play the game against the bot as black,
if no algorithm depth is specified the default (5) will be set if no algorithm depth is specified the default (5) will be set
@ -541,7 +541,7 @@ if no algorithm depth is specified the default (5) will be set
def default(color="black", algorithm_depth=5): def default(color="black", algorithm_depth=5):
"""default program function -> allows to play a game against bot (by default as black)""" """Default program function -> allows to play a game against bot (by default as black)."""
game.start_game(color, algorithm_depth) game.start_game(color, algorithm_depth)

8
Programming/EARIN/lab3/example.py
View File

@ -1,9 +1,9 @@
import tempfile
import os import os
import tempfile
import cv2 import cv2
import numpy as np
import matplotlib.pyplot as plt import matplotlib.pyplot as plt
import numpy as np
def main(): def main():
@ -32,7 +32,7 @@ def main():
image = cv2.imread(f.name) image = cv2.imread(f.name)
# show the image, provide window name first # show the image, provide window name first
cv2.imshow('visualization', image) cv2.imshow("visualization", image)
# add wait key. window waits until user presses a key and quits if the key is 'q' # add wait key. window waits until user presses a key and quits if the key is 'q'
if cv2.waitKey(0) == 113: if cv2.waitKey(0) == 113:
@ -49,5 +49,5 @@ def main():
cv2.destroyAllWindows() cv2.destroyAllWindows()
if __name__ == '__main__': if __name__ == "__main__":
main() main()

92
Programming/EARIN/lab3/main.py
View File

@ -1,19 +1,20 @@
""" """Program that optimizes Rastrigin function: file_ (x_point_value, y_point_value) =
Program that optimizes Rastrigin function: file_ (x_point_value, y_point_value) =
20 + (x_point_value^2 - 10cos(2πx)) + (y_point_value^2 - 10 cos(2πy)). 20 + (x_point_value^2 - 10cos(2πx)) + (y_point_value^2 - 10 cos(2πy)).
Using Evolutionary Strategy (μ, λ). Using Evolutionary Strategy (μ, λ).
""" """
import sys
import os import os
import time import sys
import tempfile import tempfile
import time
import cv2 import cv2
import matplotlib.pyplot as plt import matplotlib.pyplot as plt
import numpy as np import numpy as np
def rastrigin(x_argument, y_argument): def rastrigin(x_argument, y_argument):
"""Define the Rastrigin function""" """Define the Rastrigin function."""
return ( return (
20 20
+ x_argument**2 + x_argument**2
@ -23,11 +24,8 @@ def rastrigin(x_argument, y_argument):
) )
def generate( def generate(population, arguments):
population, """Run single generation."""
arguments
):
"""Run single generation"""
# Evaluate the fitness of each individual # Evaluate the fitness of each individual
fitness = np.array( fitness = np.array(
[ [
@ -37,20 +35,24 @@ def generate(
) )
# Select the top arguments["number_of_parents"] individuals # Select the top arguments["number_of_parents"] individuals
parents = population[np.argsort(fitness)[:arguments["number_of_parents"]]] parents = population[np.argsort(fitness)[: arguments["number_of_parents"]]]
# Generate the next generation of lambda individuals by recombination # Generate the next generation of lambda individuals by recombination
children = np.concatenate( children = np.concatenate(
[ [
np.random.permutation(parents) np.random.permutation(parents)
for i in range((arguments["size_of_population"] // arguments["number_of_parents"]) + 1) for i in range(
(arguments["size_of_population"] // arguments["number_of_parents"]) + 1
)
] ]
) )
children = children[:arguments["size_of_population"]] children = children[: arguments["size_of_population"]]
# Add mutation to the children # Add mutation to the children
mutation = np.random.normal( mutation = np.random.normal(
loc=0, scale=arguments["mutation_strength"], size=(arguments["size_of_population"], 2) loc=0,
scale=arguments["mutation_strength"],
size=(arguments["size_of_population"], 2),
) )
population = children + mutation population = children + mutation
return fitness, population return fitness, population
@ -60,45 +62,52 @@ def evolution_strategy(
arguments, arguments,
no_display=False, no_display=False,
): ):
"""Define the Evolutionary Strategy (μ, λ) algorithm""" """Define the Evolutionary Strategy (μ, λ) algorithm."""
# Initialize the population # Initialize the population
print_info = [] print_info = []
population = np.random.uniform( population = np.random.uniform(
low=arguments["min"], high=arguments["max"], size=(arguments["size_of_population"], 2) low=arguments["min"],
high=arguments["max"],
size=(arguments["size_of_population"], 2),
) )
summary = [] summary = []
if not no_display: if not no_display:
print_info.append( print_info.append(
(population, (
0, population,
f"""0:nop-{arguments["number_of_parents"]}:sop-{arguments["size_of_population"]}: 0,
f"""0:nop-{arguments["number_of_parents"]}:sop-{arguments["size_of_population"]}:
ms-{arguments["mutation_strength"]}:nog-{arguments["number_of_generations"]}: ms-{arguments["mutation_strength"]}:nog-{arguments["number_of_generations"]}:
min-max-{arguments["min"], arguments["max"]}:noo-{arguments["number_of_outputs"]}""", min-max-{arguments["min"], arguments["max"]}:noo-{arguments["number_of_outputs"]}""",
)) )
)
arguments["number_of_outputs"] = min( arguments["number_of_outputs"] = min(
[arguments["number_of_outputs"] - 1, arguments["number_of_generations"]]) [arguments["number_of_outputs"] - 1, arguments["number_of_generations"]]
)
# Iterate until we reach max number of generate and terminate # Iterate until we reach max number of generate and terminate
for generation_number in range(1, arguments["number_of_generations"] + 1): for generation_number in range(1, arguments["number_of_generations"] + 1):
fitness, population = generate( fitness, population = generate(population, arguments)
population, arguments)
step = ( step = (
arguments["number_of_generations"] // arguments["number_of_outputs"] arguments["number_of_generations"] // arguments["number_of_outputs"]
if arguments["number_of_generations"] % arguments["number_of_outputs"] == 0 if arguments["number_of_generations"] % arguments["number_of_outputs"] == 0
else arguments["number_of_generations"] // (arguments["number_of_outputs"] - 1) else arguments["number_of_generations"]
// (arguments["number_of_outputs"] - 1)
) )
offset = arguments["number_of_generations"] % step offset = arguments["number_of_generations"] % step
if (generation_number - offset) % step == 0 and not no_display: if (generation_number - offset) % step == 0 and not no_display:
print_info.append( print_info.append(
(population, (
generation_number, population,
f"""{generation_number}:nop_{arguments["number_of_parents"]}: generation_number,
f"""{generation_number}:nop_{arguments["number_of_parents"]}:
sop_{arguments["size_of_population"]}:ms_{arguments["mutation_strength"]}: sop_{arguments["size_of_population"]}:ms_{arguments["mutation_strength"]}:
nog_{arguments["number_of_generations"]}: nog_{arguments["number_of_generations"]}:
min_max_{arguments["min"], arguments["max"]}: min_max_{arguments["min"], arguments["max"]}:
noo_{arguments["number_of_outputs"]}""", noo_{arguments["number_of_outputs"]}""",
)) )
)
summary.append(population) summary.append(population)
# Evaluate the fitness of the final population # Evaluate the fitness of the final population
@ -121,7 +130,7 @@ def evolution_strategy(
def print_help(): def print_help():
"""Print program functionality and how to access it""" """Print program functionality and how to access it."""
print( print(
""" """
python main.py - Default functionality optimizing python main.py - Default functionality optimizing
@ -158,7 +167,7 @@ def print_help():
def get_output_bounds(x_data, y_data): def get_output_bounds(x_data, y_data):
"""Get x and y output limits for pyplot""" """Get x and y output limits for pyplot."""
# min_size = 0.2 # min_size = 0.2
min_output_size = ARGUMENTS["mutation_strength"] * 10 min_output_size = ARGUMENTS["mutation_strength"] * 10
@ -185,13 +194,8 @@ def get_output_bounds(x_data, y_data):
return x_bounds, y_bounds return x_bounds, y_bounds
def output( def output(population_output, generation_number, file_name="temp", save_results=False):
population_output, """Draw result of our function."""
generation_number,
file_name="temp",
save_results=False):
"""Draw result of our function"""
# define the visualization params # define the visualization params
colors = np.random.rand(len(population_output)) colors = np.random.rand(len(population_output))
@ -233,8 +237,7 @@ def output(
def print_summary(populations, file_name="temp_summary", save_results=False): def print_summary(populations, file_name="temp_summary", save_results=False):
"""Draw result of our function for chosen generations""" """Draw result of our function for chosen generations."""
with tempfile.NamedTemporaryFile(suffix=".png", delete=False) as file_: with tempfile.NamedTemporaryFile(suffix=".png", delete=False) as file_:
# iterate over the optimization steps # iterate over the optimization steps
# generate random 2D data - replace it with the results from your # generate random 2D data - replace it with the results from your
@ -254,12 +257,7 @@ def print_summary(populations, file_name="temp_summary", save_results=False):
# plot the data # plot the data
transparency = ind / (len(populations) - 1) transparency = ind / (len(populations) - 1)
color = [[transparency, 0, 0]] * len(pop) color = [[transparency, 0, 0]] * len(pop)
plt.scatter( plt.scatter(x_data, y_data, c=color, alpha=transparency, label=f"{ind}")
x_data,
y_data,
c=color,
alpha=transparency,
label=f"{ind}")
plt.xlim(bounds[0]) plt.xlim(bounds[0])
plt.ylim(bounds[1]) plt.ylim(bounds[1])
plt.savefig(file_.name) plt.savefig(file_.name)
@ -285,7 +283,7 @@ def print_summary(populations, file_name="temp_summary", save_results=False):
def user_input(): def user_input():
"""Handle user terminal arguments""" """Handle user terminal arguments."""
arguments = { arguments = {
"number_of_parents": 5, "number_of_parents": 5,
"size_of_population": 20, "size_of_population": 20,
@ -324,7 +322,7 @@ def user_input():
def print_output(print_info, save_results, summary): def print_output(print_info, save_results, summary):
""" Prints out population and summary plots """ """Prints out population and summary plots."""
for population, generation_number, file_name in print_info: for population, generation_number, file_name in print_info:
output(population, generation_number, file_name, save_results) output(population, generation_number, file_name, save_results)
summary_file_name = file_name summary_file_name = file_name

29
Programming/EARIN/lab4/main.py
View File

@ -1,26 +1,23 @@
""" """Program that predicts wine quality based on variant2.csv data."""
Program that predicts wine quality based on variant2.csv data
""" import matplotlib.pyplot as plt
import numpy as np
import pandas as pd import pandas as pd
import seaborn as sns import seaborn as sns
import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, accuracy_score, f1_score
from sklearn.linear_model import LogisticRegression from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, f1_score, mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
class LinearRegression: class LinearRegression:
"""Implements Linear regression method""" """Implements Linear regression method."""
def __init__(self): def __init__(self):
self.theta = None self.theta = None
def fit(self, x_values, y_values): def fit(self, x_values, y_values):
""" """Fit linear regression model to our training data."""
Fit linear regression model to our training data
"""
# Add a column of ones to X for the intercept term # Add a column of ones to X for the intercept term
x_values = np.concatenate((np.ones((x_values.shape[0], 1)), y_values), axis=1) x_values = np.concatenate((np.ones((x_values.shape[0], 1)), y_values), axis=1)
@ -30,9 +27,7 @@ class LinearRegression:
) )
def predict(self, x_values): def predict(self, x_values):
""" """Predict target values for our input data using the trained linear regression model."""
Predict target values for our input data using the trained linear regression model.
"""
# Add a column of ones to X for the intercept term # Add a column of ones to X for the intercept term
x_values = np.concatenate((np.ones((x_values.shape[0], 1)), x_values), axis=1) x_values = np.concatenate((np.ones((x_values.shape[0], 1)), x_values), axis=1)
@ -42,9 +37,7 @@ class LinearRegression:
return y_predicted return y_predicted
def score(self, x_values, y_values): def score(self, x_values, y_values):
""" """Compute the R-squared score of the linear regression model on our test data."""
Compute the R-squared score of the linear regression model on our test data.
"""
y_predicted = self.predict(x_values) y_predicted = self.predict(x_values)
ss_res = np.sum((y_values - y_predicted) ** 2) ss_res = np.sum((y_values - y_predicted) ** 2)
ss_tot = np.sum((y_values - np.mean(y_values)) ** 2) ss_tot = np.sum((y_values - np.mean(y_values)) ** 2)

21
Programming/EARIN/lab5/code/example.py
View File

@ -1,23 +1,24 @@
# numpy for loading dataset # numpy for loading dataset
import numpy as np import numpy as np
# pytorch for deep learning models # pytorch for deep learning models
import torch import torch
# nn like neural network # nn like neural network
import torch.nn as nn from torch import nn, optim
import torch.optim as optim
# Pima indians describes patient medical data and whether they had diabetes for last 5 years # Pima indians describes patient medical data and whether they had diabetes for last 5 years
# It is binary classification (they could either have diabetes 1 or not 0) # It is binary classification (they could either have diabetes 1 or not 0)
# load the file as a matrix of numbers, # load the file as a matrix of numbers,
dataset = np.loadtxt('pima-indians-diabetes.csv', delimiter=',') dataset = np.loadtxt("pima-indians-diabetes.csv", delimiter=",")
input_columns = 8 input_columns = 8
# split into input (X) -> in this case everything beside info whether patient had diabetes or not is input # split into input (X) -> in this case everything beside info whether patient had diabetes or not is input
# We are splitting data into two subsets by using NumPy slice operator : and choose first 8 columns using 0:8 slice # We are splitting data into two subsets by using NumPy slice operator : and choose first 8 columns using 0:8 slice
X = dataset[:,0:input_columns] X = dataset[:, 0:input_columns]
# and output (y) variables -> in this case we are only interested whether patient had diabetes or not as an output # and output (y) variables -> in this case we are only interested whether patient had diabetes or not as an output
# you can simplify that y = f(X) # you can simplify that y = f(X)
# We are splitting the data by using slice operator : and choosing last column # We are splitting the data by using slice operator : and choosing last column
y = dataset[:,input_columns] y = dataset[:, input_columns]
# we need to convert this data to pytorch tensors # we need to convert this data to pytorch tensors
# Pytorch usually operates on 32-bit floating point and NumPy by default uses 64 bit floating point # Pytorch usually operates on 32-bit floating point and NumPy by default uses 64 bit floating point
@ -27,6 +28,7 @@ X = torch.tensor(X, dtype=torch.float32)
# reshape is converting the output variable y from a 1-dimensional NumPy array to a 2-dimensional PyTorch tensor with a shape of (n, 1), where n is the number of samples in the dataset. # reshape is converting the output variable y from a 1-dimensional NumPy array to a 2-dimensional PyTorch tensor with a shape of (n, 1), where n is the number of samples in the dataset.
y = torch.tensor(y, dtype=torch.float32).reshape(-1, 1) y = torch.tensor(y, dtype=torch.float32).reshape(-1, 1)
# define the model # define the model
# this class is a subclass of nn.Module -> base class provided by PyTorch for building neural network models. # this class is a subclass of nn.Module -> base class provided by PyTorch for building neural network models.
class PimaClassifier(nn.Module): class PimaClassifier(nn.Module):
@ -63,6 +65,7 @@ class PimaClassifier(nn.Module):
# returns final output (0 or 1) # returns final output (0 or 1)
return x return x
# Create object from model class # Create object from model class
model = PimaClassifier() model = PimaClassifier()
print(model) print(model)
@ -72,7 +75,7 @@ print(model)
# we have input X and output y and we want the model to be as close to y as possible # we have input X and output y and we want the model to be as close to y as possible
# Since this is binary classification problem we will use "binary cross entropy" to measure the distance between # Since this is binary classification problem we will use "binary cross entropy" to measure the distance between
# our prediction and y # our prediction and y
loss_fn = nn.BCELoss() # binary cross entropy loss_fn = nn.BCELoss() # binary cross entropy
# Optimizer adjust model weights to produce better output # Optimizer adjust model weights to produce better output
# Its described as being able to tune itself to a lot of problems # Its described as being able to tune itself to a lot of problems
# inputs are: # inputs are:
@ -95,11 +98,11 @@ for epoch in range(n_epochs):
# and one for batches # and one for batches
for i in range(0, len(X), batch_size): for i in range(0, len(X), batch_size):
# Split X data into a batch with the size from batch_size # Split X data into a batch with the size from batch_size
Xbatch = X[i:i+batch_size] Xbatch = X[i : i + batch_size]
# run the model on the batch and return "batched" output # run the model on the batch and return "batched" output
y_pred = model(Xbatch) y_pred = model(Xbatch)
# Split y data into a batch with the size from batch_size # Split y data into a batch with the size from batch_size
ybatch = y[i:i+batch_size] ybatch = y[i : i + batch_size]
# Compare loss # Compare loss
loss = loss_fn(y_pred, ybatch) loss = loss_fn(y_pred, ybatch)
# optimize model # optimize model
@ -117,4 +120,4 @@ print(f"Accuracy {accuracy}")
# make class predictions with the model # make class predictions with the model
predictions = (model(X) > 0.5).int() predictions = (model(X) > 0.5).int()
for i in range(5): for i in range(5):
print('%s => %d (expected %d)' % (X[i].tolist(), predictions[i], y[i])) print("%s => %d (expected %d)" % (X[i].tolist(), predictions[i], y[i]))

182
Programming/EARIN/lab5/code/main.py
View File

@ -1,14 +1,15 @@
""" Implementation of a network analyzing MNIST dataset """ """Implementation of a network analyzing MNIST dataset."""
import torch
from torch import nn
from torch import optim
from torchvision import datasets, transforms
import matplotlib.pyplot as plt
import time import time
import matplotlib.pyplot as plt
import torch
from torch import nn, optim
from torchvision import datasets, transforms
def set_hyperparameters(): def set_hyperparameters():
""" sets hyperparameters used throughout the network """ """Sets hyperparameters used throughout the network."""
return { return {
"num_epochs": 5, "num_epochs": 5,
"init_input_size": 28 * 28, # MNIST images are 28x28 pixels "init_input_size": 28 * 28, # MNIST images are 28x28 pixels
@ -17,7 +18,7 @@ def set_hyperparameters():
def load_datasets(): def load_datasets():
""" Loads train and test dataset from MNIST """ """Loads train and test dataset from MNIST."""
train_dataset = datasets.MNIST( train_dataset = datasets.MNIST(
root="./data", train=True, transform=transforms.ToTensor(), download=True root="./data", train=True, transform=transforms.ToTensor(), download=True
) )
@ -28,7 +29,7 @@ def load_datasets():
def create_data_loaders(train_dataset, test_dataset): def create_data_loaders(train_dataset, test_dataset):
""" Create train and test data loaders """ """Create train and test data loaders."""
train_loader = torch.utils.data.DataLoader( train_loader = torch.utils.data.DataLoader(
dataset=train_dataset, batch_size=BATCH_SIZE, shuffle=True dataset=train_dataset, batch_size=BATCH_SIZE, shuffle=True
) )
@ -45,42 +46,38 @@ val_acc_values = []
def define_model(hyperparameters): def define_model(hyperparameters):
""" Define the multilayer perceptron training_parameters['model'] """ """Define the multilayer perceptron training_parameters['model']."""
# Define the multilayer perceptron model # Define the multilayer perceptron model
model = nn.Sequential() model = nn.Sequential()
model.add_module('flatten', nn.Flatten()) model.add_module("flatten", nn.Flatten())
input_size = hyperparameters['init_input_size'] input_size = hyperparameters["init_input_size"]
for i in range(NUM_HIDDEN_LAYERS): for i in range(NUM_HIDDEN_LAYERS):
model.add_module(f'linear{i}', nn.Linear(input_size, WIDTH)) model.add_module(f"linear{i}", nn.Linear(input_size, WIDTH))
model.add_module(f'relu{i}', nn.ReLU()) model.add_module(f"relu{i}", nn.ReLU())
input_size = WIDTH input_size = WIDTH
model.add_module('output', nn.Linear( model.add_module("output", nn.Linear(input_size, hyperparameters["num_classes"]))
input_size, hyperparameters['num_classes']))
return model return model
def get_optimizer(model): def get_optimizer(model):
""" Return optimizer function """ """Return optimizer function."""
if OPTIMIZER_TYPE == 'SGD': if OPTIMIZER_TYPE == "SGD":
return optim.SGD(model.parameters(), lr=LEARNING_RATE) return optim.SGD(model.parameters(), lr=LEARNING_RATE)
if OPTIMIZER_TYPE == 'SGD_Momentum': if OPTIMIZER_TYPE == "SGD_Momentum":
return optim.SGD(model.parameters(), lr=LEARNING_RATE, momentum=0.9) return optim.SGD(model.parameters(), lr=LEARNING_RATE, momentum=0.9)
if OPTIMIZER_TYPE == 'Adam': if OPTIMIZER_TYPE == "Adam":
return optim.Adam(model.parameters(), lr=LEARNING_RATE) return optim.Adam(model.parameters(), lr=LEARNING_RATE)
raise ValueError("Unsupported optimizer type!") raise ValueError("Unsupported optimizer type!")
def initial_configuration(): def initial_configuration():
""" """Perform all operations needed for training network."""
Perform all operations needed for training network
"""
# Set random seed for reproducibility # Set random seed for reproducibility
torch.manual_seed(42) torch.manual_seed(42)
hyperparameters = set_hyperparameters() hyperparameters = set_hyperparameters()
# Load MNIST dataset and apply transformations # Load MNIST dataset and apply transformations
train_dataset, test_dataset = load_datasets() train_dataset, test_dataset = load_datasets()
train_loader, test_loader = create_data_loaders( train_loader, test_loader = create_data_loaders(train_dataset, test_dataset)
train_dataset, test_dataset)
model = define_model(hyperparameters) model = define_model(hyperparameters)
# Loss function # Loss function
criterion = nn.CrossEntropyLoss() criterion = nn.CrossEntropyLoss()
@ -89,103 +86,89 @@ def initial_configuration():
return hyperparameters, train_loader, test_loader, model, criterion, optimizer return hyperparameters, train_loader, test_loader, model, criterion, optimizer
def single_train_iteration( def single_train_iteration(data, training_parameters, targets, batch_idx, epoch):
data, training_parameters, targets, batch_idx, epoch """Train network for single batch."""
):
"""
Train network for single batch
"""
# Reshape the input data # Reshape the input data
data = data.view(data.size(0), -1) data = data.view(data.size(0), -1)
# Forward pass # Forward pass
outputs = training_parameters['model'](data) outputs = training_parameters["model"](data)
loss = training_parameters['criterion'](outputs, targets) loss = training_parameters["criterion"](outputs, targets)
# Backward pass and optimization # Backward pass and optimization
training_parameters['optimizer'].zero_grad() training_parameters["optimizer"].zero_grad()
loss.backward() loss.backward()
training_parameters['optimizer'].step() training_parameters["optimizer"].step()
# Print loss value for every learning step # Print loss value for every learning step
if (batch_idx + 1) % 100 == 0: if (batch_idx + 1) % 100 == 0:
print( print(
f''' f"""
Epoch [{epoch+1}/{training_parameters['hyperparameters']["num_epochs"]}], Epoch [{epoch + 1}/{training_parameters["hyperparameters"]["num_epochs"]}],
Step [{batch_idx+1}/{len(training_parameters['loaders']['train_loader'])}], Step [{batch_idx + 1}/{len(training_parameters["loaders"]["train_loader"])}],
Loss: {loss.item():.4f} Loss: {loss.item():.4f}
''' """
) )
# Append loss value for every learning step # Append loss value for every learning step
loss_values.append(loss.item()) loss_values.append(loss.item())
return data, training_parameters['optimizer'] return data, training_parameters["optimizer"]
def set_loaders(train_loader, test_loader): def set_loaders(train_loader, test_loader):
""" """Put train and test loaders into one object."""
Put train and test loaders into one object return {"train_loader": train_loader, "test_loader": test_loader}
"""
return {
'train_loader': train_loader,
'test_loader': test_loader
}
def set_training_parameters(hyperparameters, loaders, model, criterion, optimizer): def set_training_parameters(hyperparameters, loaders, model, criterion, optimizer):
""" """Put all training parameters into one object."""
Put all training parameters into one object
"""
return { return {
'hyperparameters': hyperparameters, "hyperparameters": hyperparameters,
'loaders': { "loaders": {
'train_loader': loaders['train_loader'], "train_loader": loaders["train_loader"],
'test_loader': loaders['test_loader'] "test_loader": loaders["test_loader"],
}, },
'model': model, "model": model,
'criterion': criterion, "criterion": criterion,
'optimizer': optimizer, "optimizer": optimizer,
} }
def training_loop(training_parameters, print_info=True): def training_loop(training_parameters, print_info=True):
""" """Train network for all epochs."""
Train network for all epochs
"""
epochs_num = training_parameters["hyperparameters"]["num_epochs"] epochs_num = training_parameters["hyperparameters"]["num_epochs"]
# Training loop # Training loop
for epoch in range(epochs_num): for epoch in range(epochs_num):
for batch_idx, (data, targets) in enumerate(training_parameters['loaders']['train_loader']): for batch_idx, (data, targets) in enumerate(
data, training_parameters['optimizer'] = single_train_iteration( training_parameters["loaders"]["train_loader"]
):
data, training_parameters["optimizer"] = single_train_iteration(
data, training_parameters, targets, batch_idx, epoch data, training_parameters, targets, batch_idx, epoch
) )
calculate_accuracy_epoch( calculate_accuracy_epoch(training_parameters, epoch, print_info)
training_parameters, epoch, print_info) calculate_validation_set_accuracy(training_parameters, epoch, print_info)
calculate_validation_set_accuracy( return epoch, training_parameters["loaders"]["train_loader"]
training_parameters, epoch, print_info)
return epoch, training_parameters['loaders']['train_loader']
def calculate_accuracy_epoch(training_parameters, epoch, print_info=True): def calculate_accuracy_epoch(training_parameters, epoch, print_info=True):
""" Calculate accuracy on train set after each epoch """ """Calculate accuracy on train set after each epoch."""
correct = 0 correct = 0
total = 0 total = 0
for data, targets in training_parameters['loaders']['train_loader']: for data, targets in training_parameters["loaders"]["train_loader"]:
data = data.view(data.size(0), -1) data = data.view(data.size(0), -1)
outputs = training_parameters['model'](data) outputs = training_parameters["model"](data)
_, predicted = torch.max(outputs.data, 1) _, predicted = torch.max(outputs.data, 1)
total += targets.size(0) total += targets.size(0)
correct += (predicted == targets).sum().item() correct += (predicted == targets).sum().item()
train_accuracy = 100 * correct / total train_accuracy = 100 * correct / total
if print_info: if print_info:
print( print(f"Accuracy on Train Set after Epoch {epoch + 1}: {train_accuracy:.2f}%")
f"Accuracy on Train Set after Epoch {epoch+1}: {train_accuracy:.2f}%")
train_acc_values.append(train_accuracy) train_acc_values.append(train_accuracy)
def calculate_validation_set_accuracy(training_parameters, epoch, print_info=True): def calculate_validation_set_accuracy(training_parameters, epoch, print_info=True):
""" Calculate accuracy on validation set after each epoch """ """Calculate accuracy on validation set after each epoch."""
correct = 0 correct = 0
total = 0 total = 0
for data, targets in training_parameters['loaders']['test_loader']: for data, targets in training_parameters["loaders"]["test_loader"]:
data = data.view(data.size(0), -1) data = data.view(data.size(0), -1)
outputs = training_parameters['model'](data) outputs = training_parameters["model"](data)
_, predicted = torch.max(outputs.data, 1) _, predicted = torch.max(outputs.data, 1)
total += targets.size(0) total += targets.size(0)
correct += (predicted == targets).sum().item() correct += (predicted == targets).sum().item()
@ -193,7 +176,7 @@ def calculate_validation_set_accuracy(training_parameters, epoch, print_info=Tru
validation_accuracy = 100 * correct / total validation_accuracy = 100 * correct / total
if print_info: if print_info:
print( print(
f"Accuracy on Validation Set after Epoch {epoch+1}: {validation_accuracy:.2f}%" f"Accuracy on Validation Set after Epoch {epoch + 1}: {validation_accuracy:.2f}%"
) )
print("---") print("---")
val_acc_values.append(validation_accuracy) val_acc_values.append(validation_accuracy)
@ -209,52 +192,55 @@ def main_part(show_plot=True):
OPTIMIZER, OPTIMIZER,
) = initial_configuration() ) = initial_configuration()
start_time = time.time() start_time = time.time()
LOADERS = set_loaders( LOADERS = set_loaders(TRAIN_LOADER, TEST_LOADER)
TRAIN_LOADER, TEST_LOADER)
TRAINING_PARAMETERS = set_training_parameters( TRAINING_PARAMETERS = set_training_parameters(
HYPERPARAMETERS, LOADERS, MODEL, CRITERION, OPTIMIZER) HYPERPARAMETERS, LOADERS, MODEL, CRITERION, OPTIMIZER
)
training_loop(TRAINING_PARAMETERS, show_plot) training_loop(TRAINING_PARAMETERS, show_plot)
file = open("results.txt", "a") file = open("results.txt", "a")
file.write( file.write(
"-------------------------------------------------------------------------------------" + "\n") "-------------------------------------------------------------------------------------"
"\n"
)
file.write( file.write(
f"loss-lr{LEARNING_RATE}-bs{BATCH_SIZE}-hl{NUM_HIDDEN_LAYERS}-w{WIDTH}-{OPTIMIZER_TYPE}" + "\n") f"loss-lr{LEARNING_RATE}-bs{BATCH_SIZE}-hl{NUM_HIDDEN_LAYERS}-w{WIDTH}-{OPTIMIZER_TYPE}"
"\n"
)
file.write(f"Execution time: {(time.time() - start_time)}" + "\n") file.write(f"Execution time: {(time.time() - start_time)}" + "\n")
file.write( file.write(
"-------------------------------------------------------------------------------------" + "\n") "-------------------------------------------------------------------------------------"
"\n"
)
# Plot the loss value for every learning step # Plot the loss value for every learning step
learning_step_title = f'loss-lr{LEARNING_RATE}-bs{BATCH_SIZE}-hl{NUM_HIDDEN_LAYERS}-w{WIDTH}-{OPTIMIZER_TYPE}.png' learning_step_title = f"loss-lr{LEARNING_RATE}-bs{BATCH_SIZE}-hl{NUM_HIDDEN_LAYERS}-w{WIDTH}-{OPTIMIZER_TYPE}.png"
plt.plot(loss_values) plt.plot(loss_values)
plt.xlabel('Learning Step') plt.xlabel("Learning Step")
plt.ylabel('Loss') plt.ylabel("Loss")
plt.title(learning_step_title) plt.title(learning_step_title)
plt.savefig(learning_step_title plt.savefig(learning_step_title)
)
if show_plot: if show_plot:
plt.show() plt.show()
plt.close() plt.close()
# Plot the accuracy on train set after each epoch # Plot the accuracy on train set after each epoch
train_accuracy_title = f'trainAccuracy-lr{LEARNING_RATE}-bs{BATCH_SIZE}-hl{NUM_HIDDEN_LAYERS}-w{WIDTH}-{OPTIMIZER_TYPE}.png' train_accuracy_title = f"trainAccuracy-lr{LEARNING_RATE}-bs{BATCH_SIZE}-hl{NUM_HIDDEN_LAYERS}-w{WIDTH}-{OPTIMIZER_TYPE}.png"
plt.plot(train_acc_values) plt.plot(train_acc_values)
plt.xlabel('Epoch') plt.xlabel("Epoch")
plt.ylabel('Train Accuracy') plt.ylabel("Train Accuracy")
plt.title(train_accuracy_title) plt.title(train_accuracy_title)
plt.savefig( plt.savefig(train_accuracy_title)
train_accuracy_title)
if show_plot: if show_plot:
plt.show() plt.show()
plt.close() plt.close()
# Plot the accuracy on validation set after each epoch # Plot the accuracy on validation set after each epoch
validation_accuracy_title = f'validationAccuracy-lr{LEARNING_RATE}-bs{BATCH_SIZE}-hl{NUM_HIDDEN_LAYERS}-w{WIDTH}-{OPTIMIZER_TYPE}.png' validation_accuracy_title = f"validationAccuracy-lr{LEARNING_RATE}-bs{BATCH_SIZE}-hl{NUM_HIDDEN_LAYERS}-w{WIDTH}-{OPTIMIZER_TYPE}.png"
plt.plot(val_acc_values) plt.plot(val_acc_values)
plt.xlabel('Epoch') plt.xlabel("Epoch")
plt.ylabel('Validation Accuracy') plt.ylabel("Validation Accuracy")
plt.title(validation_accuracy_title) plt.title(validation_accuracy_title)
plt.savefig( plt.savefig(validation_accuracy_title)
validation_accuracy_title)
if show_plot: if show_plot:
plt.show() plt.show()
plt.close() plt.close()
@ -265,7 +251,7 @@ if __name__ == "__main__":
BATCH_SIZE = 64 BATCH_SIZE = 64
NUM_HIDDEN_LAYERS = 2 NUM_HIDDEN_LAYERS = 2
WIDTH = 128 WIDTH = 128
OPTIMIZER_TYPE = 'Adam' OPTIMIZER_TYPE = "Adam"
main_part(True) main_part(True)
""" """
learning_rate_values = [0.1, 0.01, 0.001] learning_rate_values = [0.1, 0.01, 0.001]

97
Programming/EARIN/lab6/main.py
View File

@ -1,7 +1,7 @@
""" """Code used to solve MountainCar-v0 gymnasium problem using Q-Learning algorithm."""
Code used to solve MountainCar-v0 gymnasium problem using Q-Learning algorithm
"""
from datetime import datetime from datetime import datetime
import gymnasium as gym import gymnasium as gym
import numpy as np import numpy as np
@ -9,58 +9,54 @@ import numpy as np
def discretize_state(state, env, first_time): def discretize_state(state, env, first_time):
# print( # print(
# f"state: {state}, state[0]: {state[0]}, env.observation_space.low: {env.observation_space.low}") # f"state: {state}, state[0]: {state[0]}, env.observation_space.low: {env.observation_space.low}")
# print(f"state[0] - env {state[0] - env.observation_space.low}") # print(f"state[0] - env {state[0] - env.observation_space.low}")
# print(f"state - env {state - env.observation_space.low}") # print(f"state - env {state - env.observation_space.low}")
if first_time: if first_time:
substract_from_state = state[0] - env.observation_space.low substract_from_state = state[0] - env.observation_space.low
else: else:
substract_from_state = state - env.observation_space.low substract_from_state = state - env.observation_space.low
discretized_state = ( discretized_state = (substract_from_state) * np.array([10, 100])
substract_from_state) * np.array([10, 100])
discretized_state = np.round(discretized_state, 0).astype(int) discretized_state = np.round(discretized_state, 0).astype(int)
return discretized_state return discretized_state
def initialize_environment(hyperparameters): def initialize_environment(hyperparameters):
""" """Initialize environment and video recording."""
Initialize environment and video recording
"""
# Initialize environment # Initialize environment
env = gym.make('MountainCar-v0', render_mode='rgb_array') env = gym.make("MountainCar-v0", render_mode="rgb_array")
# Save video # Save video
now = datetime.now() now = datetime.now()
time_string = now.strftime("%H:%M:%S") time_string = now.strftime("%H:%M:%S")
env = gym.wrappers.RecordVideo( env = gym.wrappers.RecordVideo(
env, env,
video_folder='vid', video_folder="vid",
disable_logger=True, disable_logger=True,
name_prefix=time_string, episode_trigger=lambda x: x > 600 and x % 2 == 0) name_prefix=time_string,
episode_trigger=lambda x: x > 600 and x % 2 == 0,
)
return env return env
def initialize_q_table(env): def initialize_q_table(env):
""" """Initialize "empty" Q-table."""
Initialize "empty" Q-table
"""
# Initialize Q-table # Initialize Q-table
# n_actions = env.action_space.n # Number of possible actions, should be 3 # n_actions = env.action_space.n # Number of possible actions, should be 3
# 0 accelerate left # 0 accelerate left
# 1 dont accelerate # 1 dont accelerate
# 2 accelerate to the right # 2 accelerate to the right
# q_table = np.zeros((n_actions,)) # q_table = np.zeros((n_actions,))
num_states = (env.observation_space.high - num_states = (env.observation_space.high - env.observation_space.low) * np.array(
env.observation_space.low) * np.array([10, 100]) [10, 100]
)
num_states = np.round(num_states, 0).astype(int) + 1 num_states = np.round(num_states, 0).astype(int) + 1
q_table = np.zeros((num_states[0], num_states[1], env.action_space.n)) q_table = np.zeros((num_states[0], num_states[1], env.action_space.n))
return q_table return q_table
def initialize_hyperparameters(): def initialize_hyperparameters():
""" """Initialize hyperparameters used by algorithm."""
Initialize hyperparameters used by algorithm
"""
hyperparameters = { hyperparameters = {
"learning_rate": 0.1, "learning_rate": 0.1,
"discount_factor": 0.99, "discount_factor": 0.99,
@ -70,15 +66,13 @@ def initialize_hyperparameters():
"min_max_car_position": [-1.2, 0.6], "min_max_car_position": [-1.2, 0.6],
"min_max_car_velocity": [-0.07, 0.07], "min_max_car_velocity": [-0.07, 0.07],
"goal_x": 0.5, "goal_x": 0.5,
"truncation": 200 "truncation": 200,
} }
return hyperparameters return hyperparameters
def choose_action(hyperparameters, env, q_table, discretized_state): def choose_action(hyperparameters, env, q_table, discretized_state):
""" """Choose one of 3 actions possible for the algorithm."""
Choose one of 3 actions possible for the algorithm
"""
# hyperparameters["epsilon"]-greedy exploration-exploitation tradeoff # hyperparameters["epsilon"]-greedy exploration-exploitation tradeoff
if np.random.uniform(0, 1) < hyperparameters["epsilon"]: if np.random.uniform(0, 1) < hyperparameters["epsilon"]:
action = env.action_space.sample() # Choose a random action action = env.action_space.sample() # Choose a random action
@ -89,30 +83,34 @@ def choose_action(hyperparameters, env, q_table, discretized_state):
def update_q_table(q_table, action, hyperparameters, reward): def update_q_table(q_table, action, hyperparameters, reward):
""" """Update q_table with newest reward."""
Update q_table with newest reward
"""
# Q-table update # Q-table update
q_value = q_table[action] q_value = q_table[action]
max_q_value = np.max(q_table) max_q_value = np.max(q_table)
new_q_value = (1 - hyperparameters["learning_rate"]) * q_value + \ new_q_value = (1 - hyperparameters["learning_rate"]) * q_value + hyperparameters[
hyperparameters["learning_rate"] * \ "learning_rate"
(reward + hyperparameters["discount_factor"] * max_q_value) ] * (reward + hyperparameters["discount_factor"] * max_q_value)
q_table[action] = new_q_value q_table[action] = new_q_value
return q_table return q_table
def movement(hyperparameters, env, q_table, discretized_state, total_reward=0, episode_number=0): def movement(
""" hyperparameters, env, q_table, discretized_state, total_reward=0, episode_number=0
Choose action and observe consequences ):
""" """Choose action and observe consequences."""
action = choose_action(hyperparameters, env, q_table, discretized_state) action = choose_action(hyperparameters, env, q_table, discretized_state)
# Take the action and observe the next state # Take the action and observe the next state
next_state, reward, terminated, truncated, _ = env.step(action) next_state, reward, terminated, truncated, _ = env.step(action)
discretized_next_state = discretize_state(next_state, env, False) discretized_next_state = discretize_state(next_state, env, False)
# print(discretized_next_state[0], discretized_next_state[1]) # print(discretized_next_state[0], discretized_next_state[1])
q_table[discretized_state[0], discretized_state[1], action] += hyperparameters["learning_rate"] * (reward + hyperparameters["discount_factor"] * np.max( q_table[discretized_state[0], discretized_state[1], action] += hyperparameters[
q_table[discretized_next_state[0], discretized_next_state[1]]) - q_table[discretized_state[0], discretized_state[1], action]) "learning_rate"
] * (
reward
+ hyperparameters["discount_factor"]
* np.max(q_table[discretized_next_state[0], discretized_next_state[1]])
- q_table[discretized_state[0], discretized_state[1], action]
)
total_reward += reward total_reward += reward
discretized_state = discretized_next_state discretized_state = discretized_next_state
@ -123,9 +121,7 @@ def movement(hyperparameters, env, q_table, discretized_state, total_reward=0, e
def episode_step(env, hyperparameters, q_table, episode_rewards, episode_number): def episode_step(env, hyperparameters, q_table, episode_rewards, episode_number):
""" """Actions done with every episode."""
Actions done with every episode
"""
state = env.reset() # Reset the environment to an initial state state = env.reset() # Reset the environment to an initial state
discretized_state = discretize_state(state, env, True) discretized_state = discretize_state(state, env, True)
done = False # Boolean to indicate episode completion done = False # Boolean to indicate episode completion
@ -133,7 +129,13 @@ def episode_step(env, hyperparameters, q_table, episode_rewards, episode_number)
for step in range(hyperparameters["max_steps"]): for step in range(hyperparameters["max_steps"]):
hyperparameters, env, q_table, done, discretized_state, total_reward = movement( hyperparameters, env, q_table, done, discretized_state, total_reward = movement(
hyperparameters, env, q_table, discretized_state, total_reward, episode_number) hyperparameters,
env,
q_table,
discretized_state,
total_reward,
episode_number,
)
if done: if done:
break break
@ -142,22 +144,19 @@ def episode_step(env, hyperparameters, q_table, episode_rewards, episode_number)
def training_loop(hyperparameters, env, q_table): def training_loop(hyperparameters, env, q_table):
""" """Actual training for MountainCar."""
Actual training for MountainCar
"""
episode_rewards = [] # List to store episode rewards episode_rewards = [] # List to store episode rewards
for episode_number in range(hyperparameters["max_episodes"]): for episode_number in range(hyperparameters["max_episodes"]):
env, hyperparameters, q_table, episode_rewards = episode_step( env, hyperparameters, q_table, episode_rewards = episode_step(
env, hyperparameters, q_table, episode_rewards, episode_number) env, hyperparameters, q_table, episode_rewards, episode_number
)
return env, q_table return env, q_table
def inference(env, q_table): def inference(env, q_table):
""" """Inference using the updated Q-table."""
Inference using the updated Q-table
"""
env.reset() env.reset()
done = False done = False
@ -169,7 +168,7 @@ def inference(env, q_table):
done = terminated or truncated done = terminated or truncated
if __name__ == '__main__': if __name__ == "__main__":
HYPERPARAMETERS = initialize_hyperparameters() HYPERPARAMETERS = initialize_hyperparameters()
ENV = initialize_environment(HYPERPARAMETERS) ENV = initialize_environment(HYPERPARAMETERS)
Q_TABLE = initialize_q_table(ENV) Q_TABLE = initialize_q_table(ENV)

423
Programming/EARIN/project/final/code/main.py
View File

@ -1,28 +1,25 @@
"""Code for preprocessing data and creating model that predicts and
recomends anime based on another anime entered by user.
""" """
Code for preprocessing data and creating model that predicts and
recomends anime based on another anime entered by user
"""
import math
import argparse import argparse
import shutil
import os
import datetime import datetime
import pandas as pd import math
import os
import shutil
import numpy as np import numpy as np
from sklearn.neighbors import NearestNeighbors import pandas as pd
from sklearn.neighbors import VALID_METRICS_SPARSE
from scipy.sparse import csr_matrix from scipy.sparse import csr_matrix
from sklearn.neighbors import NearestNeighbors
def get_data_cpu(limit_data=-1, data_folder_path="database"): def get_data_cpu(limit_data=-1, data_folder_path="database"):
""" """Reads anime from csv database."""
Reads anime from csv database
"""
if limit_data > -1: if limit_data > -1:
# User can limit number of data taken into consideration, # User can limit number of data taken into consideration,
# model seems to work with limit_data value as low as 500,000 # model seems to work with limit_data value as low as 500,000
rating_data = pd.read_csv( rating_data = pd.read_csv(data_folder_path + "/animelist.csv", nrows=limit_data)
data_folder_path + "/animelist.csv", nrows=limit_data)
else: else:
rating_data = pd.read_csv(data_folder_path + "/animelist.csv") rating_data = pd.read_csv(data_folder_path + "/animelist.csv")
anime_data = pd.read_csv(data_folder_path + "/anime.csv") anime_data = pd.read_csv(data_folder_path + "/anime.csv")
@ -40,15 +37,12 @@ def get_data(limit_data=-1, data_folder_path="database", gpu=False):
def merge_rating_anime_data(rating_data, anime_contact_data, debug=False): def merge_rating_anime_data(rating_data, anime_contact_data, debug=False):
""" """Preprocesses the data used for rating."""
Preprocesses the data used for rating
"""
rating_data = rating_data.merge( rating_data = rating_data.merge(
anime_contact_data, left_on="anime_id", right_on="anime_id", how="left" anime_contact_data, left_on="anime_id", right_on="anime_id", how="left"
) )
rating_data = rating_data[ rating_data = rating_data[
["user_id", "Name", "anime_id", "rating", ["user_id", "Name", "anime_id", "rating", "watching_status", "watched_episodes"]
"watching_status", "watched_episodes"]
] ]
rating_head = rating_data.head() rating_head = rating_data.head()
if debug: if debug:
@ -60,9 +54,7 @@ def merge_rating_anime_data(rating_data, anime_contact_data, debug=False):
def split_data_below_thresholds(rating_data, data_name, threshold=-1, debug=False): def split_data_below_thresholds(rating_data, data_name, threshold=-1, debug=False):
""" """Removes data with data_name which is below given threshold."""
Removes data with data_name which is below given threshold
"""
if threshold != -1: if threshold != -1:
count = rating_data[data_name].value_counts() count = rating_data[data_name].value_counts()
rating_data = rating_data[ rating_data = rating_data[
@ -75,9 +67,8 @@ def split_data_below_thresholds(rating_data, data_name, threshold=-1, debug=Fals
def combine_name_and_ratings(rating_data, debug=False): def combine_name_and_ratings(rating_data, debug=False):
""" """Create table which holds name of the anime and number of its reviews
Create table which holds name of the anime and number of its reviews then we merge this with rating_data.
then we merge this with rating_data
""" """
combine_movie_rating = rating_data.dropna(axis=0, subset=["Name"]) combine_movie_rating = rating_data.dropna(axis=0, subset=["Name"])
movie_rating_count = ( movie_rating_count = (
@ -95,44 +86,39 @@ def combine_name_and_ratings(rating_data, debug=False):
def get_length_of_data(rating_data, data_name): def get_length_of_data(rating_data, data_name):
""" """We get amount of data in the database with a given column data_name."""
We get amount of data in the database with a given column data_name
"""
# Encoding categorical data # Encoding categorical data
column_ids = rating_data[data_name + "_id"].unique().tolist() column_ids = rating_data[data_name + "_id"].unique().tolist()
column_to_column = {x: i for i, x in enumerate(column_ids)} column_to_column = {x: i for i, x in enumerate(column_ids)}
rating_data[data_name] = rating_data[data_name + rating_data[data_name] = rating_data[data_name + "_id"].map(column_to_column)
"_id"].map(column_to_column)
users_number = len(column_to_column) users_number = len(column_to_column)
return users_number return users_number
def get_top_ranked(rating_data, data_name, join_table=None, top_data_taken=20): def get_top_ranked(rating_data, data_name, join_table=None, top_data_taken=20):
""" """Get anime with highest ranking."""
Get anime with highest ranking
"""
if join_table is None: if join_table is None:
join_table = rating_data join_table = rating_data
group_data_by_rating = rating_data.groupby( group_data_by_rating = rating_data.groupby(data_name + "_id")["rating"].count()
data_name + "_id")["rating"].count()
top_users = group_data_by_rating.dropna().sort_values(ascending=False)[ top_users = group_data_by_rating.dropna().sort_values(ascending=False)[
:top_data_taken] :top_data_taken
top_rated = join_table.join(top_users, rsuffix="_r", ]
how="inner", on=data_name + "_id") top_rated = join_table.join(
top_users, rsuffix="_r", how="inner", on=data_name + "_id"
)
return top_rated return top_rated
def get_data_info(rating_data, debug=False, gpu=False): def get_data_info(rating_data, debug=False, gpu=False):
""" """Get some informations about data."""
Get some informations about data
"""
users_number = get_length_of_data(rating_data, "user") users_number = get_length_of_data(rating_data, "user")
animes_number = get_length_of_data(rating_data, "anime") animes_number = get_length_of_data(rating_data, "anime")
top_rated = get_top_ranked(rating_data, "user") top_rated = get_top_ranked(rating_data, "user")
top_rated = get_top_ranked(rating_data, "anime", top_rated) top_rated = get_top_ranked(rating_data, "anime", top_rated)
pivot = pd.crosstab(top_rated.user_id, top_rated.anime_id, pivot = pd.crosstab(
top_rated.rating, aggfunc=np.sum) top_rated.user_id, top_rated.anime_id, top_rated.rating, aggfunc=np.sum
)
pivot.fillna(0, inplace=True) pivot.fillna(0, inplace=True)
smallest_rating = min(rating_data["rating"]) smallest_rating = min(rating_data["rating"])
@ -142,19 +128,22 @@ def get_data_info(rating_data, debug=False, gpu=False):
if debug: if debug:
print(f"Num of users: {users_number}, Num of animes: {animes_number}") print(f"Num of users: {users_number}, Num of animes: {animes_number}")
print( print(
f"Min total rating: {smallest_rating}, Max total rating: {highest_rating}") f"Min total rating: {smallest_rating}, Max total rating: {highest_rating}"
)
def preprocessing(rating_data, anime_contact_data, def preprocessing(
debug=False, user_threshold=500, anime_threshold=200, auto=False): rating_data,
""" anime_contact_data,
Preprocesses data for making model more accurate and/or faster debug=False,
""" user_threshold=500,
anime_threshold=200,
auto=False,
):
"""Preprocesses data for making model more accurate and/or faster."""
rating_data = merge_rating_anime_data(rating_data, anime_contact_data) rating_data = merge_rating_anime_data(rating_data, anime_contact_data)
rating_data = split_data_below_thresholds( rating_data = split_data_below_thresholds(rating_data, "user_id", user_threshold)
rating_data, "user_id", user_threshold) rating_data = split_data_below_thresholds(rating_data, "anime_id", anime_threshold)
rating_data = split_data_below_thresholds(
rating_data, "anime_id", anime_threshold)
rating_data = combine_name_and_ratings(rating_data) rating_data = combine_name_and_ratings(rating_data)
rating_data = rating_data.drop(columns="rating_y") rating_data = rating_data.drop(columns="rating_y")
@ -171,10 +160,16 @@ def preprocessing(rating_data, anime_contact_data,
return pivot_table return pivot_table
def predict(prediction_model, pivot_table, seed=42, anime="RANDOM", recommendation_number=6, auto=False, debug=False): def predict(
""" prediction_model,
This will choose a random anime name and our prediction_model will predict similar anime. pivot_table,
""" seed=42,
anime="RANDOM",
recommendation_number=6,
auto=False,
debug=False,
):
"""This will choose a random anime name and our prediction_model will predict similar anime."""
np.random.seed(seed) np.random.seed(seed)
if anime == "RANDOM": if anime == "RANDOM":
chosen_anime = np.random.choice(pivot_table.shape[0]) chosen_anime = np.random.choice(pivot_table.shape[0])
@ -183,11 +178,10 @@ def predict(prediction_model, pivot_table, seed=42, anime="RANDOM", recommendati
else: else:
query = pivot_table.loc[anime].values.reshape(1, -1) query = pivot_table.loc[anime].values.reshape(1, -1)
chosen_anime_name = anime chosen_anime_name = anime
distance, suggestions = prediction_model.kneighbors( distance, suggestions = prediction_model.kneighbors(query)
query)
if debug: if debug:
print("prediction model, distance: ", distance) print("prediction model, distance: ", distance)
for i in range(0, 2): for i in range(2):
if i == 0: if i == 0:
print(f"Recommendations for {chosen_anime_name}:\n") print(f"Recommendations for {chosen_anime_name}:\n")
else: else:
@ -200,9 +194,15 @@ def predict(prediction_model, pivot_table, seed=42, anime="RANDOM", recommendati
closest_anime_distance = distance.flatten()[1] closest_anime_distance = distance.flatten()[1]
average_minus_closest_distance = average_distance - closest_anime_distance average_minus_closest_distance = average_distance - closest_anime_distance
print( print(
f"Average distance: {average_distance}, average_minus_closest_distance: {average_minus_closest_distance}") f"Average distance: {average_distance}, average_minus_closest_distance: {average_minus_closest_distance}"
)
return chosen_anime, suggestions.flatten()[1:recommendation_number+1], distance.flatten()[1:recommendation_number+1], f"{closest_anime_distance}_{average_distance}_{average_minus_closest_distance}" return (
chosen_anime,
suggestions.flatten()[1 : recommendation_number + 1],
distance.flatten()[1 : recommendation_number + 1],
f"{closest_anime_distance}_{average_distance}_{average_minus_closest_distance}",
)
# return f"{chosen_anime_name}_{closest_anime_name}_{closest_anime_distance}_{average_distance}_{average_minus_closest_distance}" # return f"{chosen_anime_name}_{closest_anime_name}_{closest_anime_distance}_{average_distance}_{average_minus_closest_distance}"
@ -212,25 +212,25 @@ def calculate_neighbors(rows_number, neighbors=5):
"sqrt": math.floor(math.sqrt(rows_number)), "sqrt": math.floor(math.sqrt(rows_number)),
"half": math.floor(rows_number / 2), "half": math.floor(rows_number / 2),
"log": math.floor(math.log(rows_number)), "log": math.floor(math.log(rows_number)),
"n-1": rows_number - 1 "n-1": rows_number - 1,
} }
if isinstance(neighbors, str): if isinstance(neighbors, str):
return neighbor_value[neighbors] return neighbor_value[neighbors]
return neighbors return neighbors
def create_model(pivot_table, rows_number, metric="cosine", algorithm="brute", neighbors=5): def create_model(
""" pivot_table, rows_number, metric="cosine", algorithm="brute", neighbors=5
Creates model based on neaarest neighbor for anime prediction ):
""" """Creates model based on neaarest neighbor for anime prediction."""
neighbors_number = calculate_neighbors(pivot_table.shape[0], neighbors) neighbors_number = calculate_neighbors(pivot_table.shape[0], neighbors)
pivot_table_matrix = csr_matrix(pivot_table.values) pivot_table_matrix = csr_matrix(pivot_table.values)
if algorithm == "brute": if algorithm == "brute":
model = NearestNeighbors(n_neighbors=neighbors_number,
metric=metric, algorithm=algorithm)
else:
model = NearestNeighbors( model = NearestNeighbors(
n_neighbors=neighbors_number, algorithm=algorithm) n_neighbors=neighbors_number, metric=metric, algorithm=algorithm
)
else:
model = NearestNeighbors(n_neighbors=neighbors_number, algorithm=algorithm)
try: try:
model.fit(pivot_table_matrix) model.fit(pivot_table_matrix)
except: except:
@ -241,103 +241,180 @@ def create_model(pivot_table, rows_number, metric="cosine", algorithm="brute", n
def handle_arguments(): def handle_arguments():
""" """Handles all arguments that can be used to change algorithm behaviour or program display."""
Handles all arguments that can be used to change algorithm behaviour or program display parser = argparse.ArgumentParser(description="Example script with pyargs")
""" parser.add_argument(
parser = argparse.ArgumentParser(description='Example script with pyargs') "--data_limit",
parser.add_argument('--data_limit', '-dl', "-dl",
help="""Specify data limit, help="""Specify data limit,
Recommended at least 500k, set to -1 for no limit""", Recommended at least 500k, set to -1 for no limit""",
required=False, type=int, default=-1) required=False,
parser.add_argument('--seed', '-s', type=int,
help='Specify seed', default=-1,
type=int, required=False, default=42) )
parser.add_argument('--debug', '-d', parser.add_argument(
help='Use debug (more information) prints', "--seed", "-s", help="Specify seed", type=int, required=False, default=42
type=bool, required=False, default=False) )
parser.add_argument('--database', '-db', parser.add_argument(
help='Specify database path', "--debug",
required=False, default="database") "-d",
help="Use debug (more information) prints",
type=bool,
required=False,
default=False,
)
parser.add_argument(
"--database",
"-db",
help="Specify database path",
required=False,
default="database",
)
allowed_metric = ["cosine", "mahalanobis", "euclidean"] allowed_metric = ["cosine", "mahalanobis", "euclidean"]
parser.add_argument('--metric', '-m', parser.add_argument(
help='Specify metric for NearestNeighbor learner', "--metric",
required=False, default="cosine", choices=allowed_metric) "-m",
allowed_algorithms = ['auto', 'brute'] help="Specify metric for NearestNeighbor learner",
parser.add_argument('--algorithm', '-a', required=False,
help='Specify algorithm for Nearest Neighbor learner', default="cosine",
required=False, default="brute", choices=allowed_algorithms) choices=allowed_metric,
parser.add_argument('--anime', '-an', )
help='Specify anime to choose', allowed_algorithms = ["auto", "brute"]
required=False, default="RANDOM") parser.add_argument(
parser.add_argument('--neighbors', '-n', "--algorithm",
help='Specify number of nearest neighbors', "-a",
required=False, default=5) help="Specify algorithm for Nearest Neighbor learner",
parser.add_argument('--user_threshold', '-ut', required=False,
help="""Specify minimal number of votes required for user to be default="brute",
choices=allowed_algorithms,
)
parser.add_argument(
"--anime",
"-an",
help="Specify anime to choose",
required=False,
default="RANDOM",
)
parser.add_argument(
"--neighbors",
"-n",
help="Specify number of nearest neighbors",
required=False,
default=5,
)
parser.add_argument(
"--user_threshold",
"-ut",
help="""Specify minimal number of votes required for user to be
included in the data, set to -1 for no threshold""", included in the data, set to -1 for no threshold""",
required=False, type=int, default=500) required=False,
parser.add_argument('--anime_threshold', '-at', type=int,
help="""Specify minimal number of votes required for anime default=500,
)
parser.add_argument(
"--anime_threshold",
"-at",
help="""Specify minimal number of votes required for anime
to be included in the data, set to -1 for no threshold""", to be included in the data, set to -1 for no threshold""",
required=False, type=int, default=200) required=False,
parser.add_argument('--recommendation_amount', '-ra', type=int,
help='Specify how much anime should be recommended', default=200,
required=False, type=int, default=5) )
parser.add_argument('--auto', '-au', parser.add_argument(
help="""Enable auto mode, no debug, no user parameters, "--recommendation_amount",
"-ra",
help="Specify how much anime should be recommended",
required=False,
type=int,
default=5,
)
parser.add_argument(
"--auto",
"-au",
help="""Enable auto mode, no debug, no user parameters,
automatic testing and saving results""", automatic testing and saving results""",
type=bool, required=False, default=False) type=bool,
required=False,
default=False,
)
# Parse the command-line arguments # Parse the command-line arguments
args = parser.parse_args() args = parser.parse_args()
args.recommendation_amount = args.recommendation_amount + 1 args.recommendation_amount = args.recommendation_amount + 1
# Access the values of the arguments # Access the values of the arguments
return args.seed, args.debug, args.data_limit, args.database, args.metric, args.algorithm, args.anime, args.neighbors, args.user_threshold, args.anime_threshold, args.recommendation_amount, args.auto return (
args.seed,
args.debug,
args.data_limit,
args.database,
args.metric,
args.algorithm,
args.anime,
args.neighbors,
args.user_threshold,
args.anime_threshold,
args.recommendation_amount,
args.auto,
)
def auto_mode(data_limit=-1, seed=42, anime="RANDOM"): def auto_mode(data_limit=-1, seed=42, anime="RANDOM"):
print("Started auto mode") print("Started auto mode")
algorithm_spread = ['auto', 'brute'] algorithm_spread = ["auto", "brute"]
metric_spread = ['manhattan', 'euclidean', 'cosine'] metric_spread = ["manhattan", "euclidean", "cosine"]
neighbor_spread = [5, "sqrt", "half", "log", "n-1"] neighbor_spread = [5, "sqrt", "half", "log", "n-1"]
# No reason to access and waste computational power every time we run the simulation # No reason to access and waste computational power every time we run the simulation
starting_rating_data, starting_anime_contact_data, starting_rows_number = get_data( starting_rating_data, starting_anime_contact_data, starting_rows_number = get_data(
limit_data=data_limit) limit_data=data_limit
)
original_pivot_table = preprocessing( original_pivot_table = preprocessing(
starting_rating_data, starting_anime_contact_data) starting_rating_data, starting_anime_contact_data
if os.path.exists('test_results'): )
shutil.rmtree('test_results') if os.path.exists("test_results"):
shutil.rmtree("test_results")
for algorithm in algorithm_spread: for algorithm in algorithm_spread:
possibleMetrics = [] possibleMetrics = []
if algorithm != 'auto': if algorithm != "auto":
possibleMetrics = metric_spread possibleMetrics = metric_spread
print("testing for algorithm: ", algorithm, possibleMetrics) print("testing for algorithm: ", algorithm, possibleMetrics)
if possibleMetrics == []: if possibleMetrics == []:
possibleMetrics = [""] possibleMetrics = [""]
for metric in possibleMetrics: for metric in possibleMetrics:
if metric != 'precomputed': if metric != "precomputed":
print("testing for algorithm, metric: ", algorithm, metric) print("testing for algorithm, metric: ", algorithm, metric)
for neighbor_amount in neighbor_spread: for neighbor_amount in neighbor_spread:
print("testing for algorithm, metric, neighbor_amount: ", print(
algorithm, metric, neighbor_amount) "testing for algorithm, metric, neighbor_amount: ",
preprocess_model_predict(starting_rating_data, starting_anime_contact_data, algorithm,
starting_rows_number, original_pivot_table, seed=seed, anime=anime, neighbors=neighbor_amount, algorithm=algorithm, metric=metric) metric,
neighbor_amount,
)
preprocess_model_predict(
starting_rating_data,
starting_anime_contact_data,
starting_rows_number,
original_pivot_table,
seed=seed,
anime=anime,
neighbors=neighbor_amount,
algorithm=algorithm,
metric=metric,
)
def write_test_results(title, result=""): def write_test_results(title, result=""):
# Create directory if it doesn't already exist # Create directory if it doesn't already exist
if not os.path.exists('test_results'): if not os.path.exists("test_results"):
os.makedirs('test_results') os.makedirs("test_results")
# Generate timestamped filename # Generate timestamped filename
timestamp = datetime.datetime.now().strftime( timestamp = datetime.datetime.now().strftime("%Y%m%d%H%M%S") # e.g., 20230611235959
'%Y%m%d%H%M%S') # e.g., 20230611235959
filename = f"{title}_{timestamp}.txt" filename = f"{title}_{timestamp}.txt"
# Create and write to the file # Create and write to the file
with open(os.path.join('test_results', filename), 'a') as file: with open(os.path.join("test_results", filename), "a") as file:
file.write(result) file.write(result)
@ -347,42 +424,94 @@ def calculate_precision(predictions, threshold=8):
return np.mean(precision) return np.mean(precision)
def preprocess_model_predict(rating_data, anime_contact_data, rows_number, pivot_table, data_limit=-1, db="database", debug=False, user_threshold=500, anime_threshold=200, metric="cosine", algorithm="brute", neighbors=5, seed=42, anime="RANDOM", recommendation_amount=5): def preprocess_model_predict(
MODEL = create_model(pivot_table, rows_number, rating_data,
metric, algorithm, neighbors) anime_contact_data,
rows_number,
pivot_table,
data_limit=-1,
db="database",
debug=False,
user_threshold=500,
anime_threshold=200,
metric="cosine",
algorithm="brute",
neighbors=5,
seed=42,
anime="RANDOM",
recommendation_amount=5,
):
MODEL = create_model(pivot_table, rows_number, metric, algorithm, neighbors)
result = "" result = ""
if MODEL != "Error!": if MODEL != "Error!":
chosen_anime, suggestions, distance, distance_data = predict(MODEL, pivot_table, seed, chosen_anime, suggestions, distance, distance_data = predict(
anime, recommendation_amount) MODEL, pivot_table, seed, anime, recommendation_amount
)
chosen_anime_name = pivot_table.index[chosen_anime] chosen_anime_name = pivot_table.index[chosen_anime]
# average_distance = np.mean(distance) # average_distance = np.mean(distance)
# closest_anime_name = pivot_table.index[suggestions[1]] # closest_anime_name = pivot_table.index[suggestions[1]]
# closest_anime_distance = distance[1] # closest_anime_distance = distance[1]
# average_minus_closest_distance = closest_anime_distance - average_distance # average_minus_closest_distance = closest_anime_distance - average_distance
precision = calculate_precision( precision = calculate_precision([pivot_table.iloc[s] for s in suggestions])
[pivot_table.iloc[s] for s in suggestions])
result = f"{chosen_anime_name}:\n" result = f"{chosen_anime_name}:\n"
for i in range(len(suggestions)): for i in range(len(suggestions)):
result += f"{pivot_table.index[suggestions[i]]}; Distance: {distance[i]}\n" result += f"{pivot_table.index[suggestions[i]]}; Distance: {distance[i]}\n"
result += f"Precision: {precision*100}%\n" result += f"Precision: {precision * 100}%\n"
result += "Smallest distance, average distance, Average - Smallest distance: " + distance_data result += (
"Smallest distance, average distance, Average - Smallest distance: "
+ distance_data
)
# result = f"{chosen_anime_name}_{closest_anime_name}_{closest_anime_distance}_{average_distance}_{average_minus_closest_distance}" # result = f"{chosen_anime_name}_{closest_anime_name}_{closest_anime_distance}_{average_distance}_{average_minus_closest_distance}"
write_test_results( write_test_results(
f"dl={rows_number}&s={seed}&m={metric}&a={algorithm}&ut={user_threshold}&at={anime_threshold}&n={neighbors}", result) f"dl={rows_number}&s={seed}&m={metric}&a={algorithm}&ut={user_threshold}&at={anime_threshold}&n={neighbors}",
result,
)
if __name__ == "__main__": if __name__ == "__main__":
SEED, DEBUG, DATA_LIMIT, DB, METRIC, ALGORITHM, ANIME, NEIGHBORS, USER_THRESHOLD, ANIME_THRESHOLD, RECOMMENDATION_AMOUNT, AUTO = handle_arguments() (
SEED,
DEBUG,
DATA_LIMIT,
DB,
METRIC,
ALGORITHM,
ANIME,
NEIGHBORS,
USER_THRESHOLD,
ANIME_THRESHOLD,
RECOMMENDATION_AMOUNT,
AUTO,
) = handle_arguments()
if not AUTO: if not AUTO:
print("Entered not auto mode") print("Entered not auto mode")
starting_rating_data, starting_anime_contact_data, starting_rows_number = get_data( starting_rating_data, starting_anime_contact_data, starting_rows_number = (
limit_data=DATA_LIMIT, data_folder_path=DB) get_data(limit_data=DATA_LIMIT, data_folder_path=DB)
)
pivot_table = preprocessing( pivot_table = preprocessing(
starting_rating_data, starting_anime_contact_data, USER_THRESHOLD, ANIME_THRESHOLD) starting_rating_data,
preprocess_model_predict(starting_rating_data, starting_anime_contact_data, starting_rows_number, starting_anime_contact_data,
pivot_table, data_limit=DATA_LIMIT, db=DB, debug=DEBUG, user_threshold=USER_THRESHOLD, anime_threshold=ANIME_THRESHOLD, USER_THRESHOLD,
metric=METRIC, algorithm=ALGORITHM, neighbors=NEIGHBORS, seed=SEED, anime=ANIME, recommendation_amount=RECOMMENDATION_AMOUNT) ANIME_THRESHOLD,
)
preprocess_model_predict(
starting_rating_data,
starting_anime_contact_data,
starting_rows_number,
pivot_table,
data_limit=DATA_LIMIT,
db=DB,
debug=DEBUG,
user_threshold=USER_THRESHOLD,
anime_threshold=ANIME_THRESHOLD,
metric=METRIC,
algorithm=ALGORITHM,
neighbors=NEIGHBORS,
seed=SEED,
anime=ANIME,
recommendation_amount=RECOMMENDATION_AMOUNT,
)
if AUTO: if AUTO:
auto_mode(DATA_LIMIT, SEED, ANIME) auto_mode(DATA_LIMIT, SEED, ANIME)

244
Programming/EARIN/project/midterm/code/main.py
View File

@ -1,26 +1,21 @@
"""Code for preprocessing data and creating model that predicts and
recomends anime based on another anime entered by user.
""" """
Code for preprocessing data and creating model that predicts and
recomends anime based on another anime entered by user
"""
import pandas as pd
import numpy as np
import argparse import argparse
import sklearn import numpy as np
from sklearn.neighbors import NearestNeighbors import pandas as pd
from scipy.sparse import csr_matrix from scipy.sparse import csr_matrix
from sklearn.neighbors import NearestNeighbors
def get_data(limit_data=-1, data_folder_path="database"): def get_data(limit_data=-1, data_folder_path="database"):
""" """Reads anime from csv database."""
Reads anime from csv database
"""
if limit_data > -1: if limit_data > -1:
# User can limit number of data taken into consideration, # User can limit number of data taken into consideration,
# model seems to work with limit_data value as low as 500,000 # model seems to work with limit_data value as low as 500,000
rating_data = pd.read_csv( rating_data = pd.read_csv(data_folder_path + "/animelist.csv", nrows=limit_data)
data_folder_path + "/animelist.csv", nrows=limit_data)
else: else:
rating_data = pd.read_csv(data_folder_path + "/animelist.csv") rating_data = pd.read_csv(data_folder_path + "/animelist.csv")
anime_data = pd.read_csv(data_folder_path + "/anime.csv") anime_data = pd.read_csv(data_folder_path + "/anime.csv")
@ -32,15 +27,12 @@ def get_data(limit_data=-1, data_folder_path="database"):
def merge_rating_anime_data(rating_data, anime_contact_data, debug=False): def merge_rating_anime_data(rating_data, anime_contact_data, debug=False):
""" """Preprocesses the data used for rating."""
Preprocesses the data used for rating
"""
rating_data = rating_data.merge( rating_data = rating_data.merge(
anime_contact_data, left_on="anime_id", right_on="anime_id", how="left" anime_contact_data, left_on="anime_id", right_on="anime_id", how="left"
) )
rating_data = rating_data[ rating_data = rating_data[
["user_id", "Name", "anime_id", "rating", ["user_id", "Name", "anime_id", "rating", "watching_status", "watched_episodes"]
"watching_status", "watched_episodes"]
] ]
rating_head = rating_data.head() rating_head = rating_data.head()
if debug: if debug:
@ -52,9 +44,7 @@ def merge_rating_anime_data(rating_data, anime_contact_data, debug=False):
def split_data_below_thresholds(rating_data, data_name, threshold=-1, debug=False): def split_data_below_thresholds(rating_data, data_name, threshold=-1, debug=False):
""" """Removes data with data_name which is below given threshold."""
Removes data with data_name which is below given threshold
"""
if threshold != -1: if threshold != -1:
count = rating_data[data_name].value_counts() count = rating_data[data_name].value_counts()
rating_data = rating_data[ rating_data = rating_data[
@ -67,9 +57,8 @@ def split_data_below_thresholds(rating_data, data_name, threshold=-1, debug=Fals
def combine_name_and_ratings(rating_data, debug=False): def combine_name_and_ratings(rating_data, debug=False):
""" """Create table which holds name of the anime and number of its reviews
Create table which holds name of the anime and number of its reviews then we merge this with rating_data.
then we merge this with rating_data
""" """
combine_movie_rating = rating_data.dropna(axis=0, subset=["Name"]) combine_movie_rating = rating_data.dropna(axis=0, subset=["Name"])
movie_rating_count = ( movie_rating_count = (
@ -87,45 +76,40 @@ def combine_name_and_ratings(rating_data, debug=False):
def get_length_of_data(rating_data, data_name): def get_length_of_data(rating_data, data_name):
""" """We get amount of data in the database with a given column data_name."""
We get amount of data in the database with a given column data_name
"""
# Encoding categorical data # Encoding categorical data
column_ids = rating_data[data_name + "_id"].unique().tolist() column_ids = rating_data[data_name + "_id"].unique().tolist()
column_to_column = {x: i for i, x in enumerate(column_ids)} column_to_column = {x: i for i, x in enumerate(column_ids)}
rating_data[data_name] = rating_data[data_name + rating_data[data_name] = rating_data[data_name + "_id"].map(column_to_column)
"_id"].map(column_to_column)
users_number = len(column_to_column) users_number = len(column_to_column)
return users_number return users_number
def get_top_ranked(rating_data, data_name, join_table=None, top_data_taken=20): def get_top_ranked(rating_data, data_name, join_table=None, top_data_taken=20):
""" """Get anime with highest ranking."""
Get anime with highest ranking
"""
if join_table is None: if join_table is None:
join_table = rating_data join_table = rating_data
group_data_by_rating = rating_data.groupby( group_data_by_rating = rating_data.groupby(data_name + "_id")["rating"].count()
data_name + "_id")["rating"].count()
top_users = group_data_by_rating.dropna().sort_values(ascending=False)[ top_users = group_data_by_rating.dropna().sort_values(ascending=False)[
:top_data_taken] :top_data_taken
top_rated = join_table.join(top_users, rsuffix="_r", ]
how="inner", on=data_name + "_id") top_rated = join_table.join(
top_users, rsuffix="_r", how="inner", on=data_name + "_id"
)
return top_rated return top_rated
def get_data_info(rating_data, debug=False): def get_data_info(rating_data, debug=False):
""" """Get some informations about data."""
Get some informations about data
"""
users_number = get_length_of_data(rating_data, "user") users_number = get_length_of_data(rating_data, "user")
animes_number = get_length_of_data(rating_data, "anime") animes_number = get_length_of_data(rating_data, "anime")
top_rated = get_top_ranked(rating_data, "user") top_rated = get_top_ranked(rating_data, "user")
top_rated = get_top_ranked(rating_data, "anime", top_rated) top_rated = get_top_ranked(rating_data, "anime", top_rated)
pivot = pd.crosstab(top_rated.user_id, top_rated.anime_id, pivot = pd.crosstab(
top_rated.rating, aggfunc=np.sum) top_rated.user_id, top_rated.anime_id, top_rated.rating, aggfunc=np.sum
)
pivot.fillna(0, inplace=True) pivot.fillna(0, inplace=True)
smallest_rating = min(rating_data["rating"]) smallest_rating = min(rating_data["rating"])
@ -135,18 +119,21 @@ def get_data_info(rating_data, debug=False):
if debug: if debug:
print(f"Num of users: {users_number}, Num of animes: {animes_number}") print(f"Num of users: {users_number}, Num of animes: {animes_number}")
print( print(
f"Min total rating: {smallest_rating}, Max total rating: {highest_rating}") f"Min total rating: {smallest_rating}, Max total rating: {highest_rating}"
)
def preprocessing(rating_data, anime_contact_data, debug=False, user_threshold=500, anime_threshold=200): def preprocessing(
""" rating_data,
Preprocesses data for making model more accurate and/or faster anime_contact_data,
""" debug=False,
user_threshold=500,
anime_threshold=200,
):
"""Preprocesses data for making model more accurate and/or faster."""
rating_data = merge_rating_anime_data(rating_data, anime_contact_data) rating_data = merge_rating_anime_data(rating_data, anime_contact_data)
rating_data = split_data_below_thresholds( rating_data = split_data_below_thresholds(rating_data, "user_id", user_threshold)
rating_data, "user_id", user_threshold) rating_data = split_data_below_thresholds(rating_data, "anime_id", anime_threshold)
rating_data = split_data_below_thresholds(
rating_data, "anime_id", anime_threshold)
rating_data = combine_name_and_ratings(rating_data) rating_data = combine_name_and_ratings(rating_data)
rating_data = rating_data.drop(columns="rating_x") rating_data = rating_data.drop(columns="rating_x")
@ -163,10 +150,10 @@ def preprocessing(rating_data, anime_contact_data, debug=False, user_threshold=5
return pivot_table return pivot_table
def predict(prediction_model, pivot_table, seed=42, anime="RANDOM", recommendation_number=6): def predict(
""" prediction_model, pivot_table, seed=42, anime="RANDOM", recommendation_number=6
This will choose a random anime name and our prediction_model will predict similar anime. ):
""" """This will choose a random anime name and our prediction_model will predict similar anime."""
np.random.seed(seed) np.random.seed(seed)
print(pivot_table) print(pivot_table)
if anime == "RANDOM": if anime == "RANDOM":
@ -178,8 +165,9 @@ def predict(prediction_model, pivot_table, seed=42, anime="RANDOM", recommendati
chosen_anime_name = anime chosen_anime_name = anime
distance, suggestions = prediction_model.kneighbors( distance, suggestions = prediction_model.kneighbors(
query, n_neighbors=recommendation_number) query, n_neighbors=recommendation_number
for i in range(0, len(distance.flatten())): )
for i in range(len(distance.flatten())):
if i == 0: if i == 0:
print(f"Recommendations for {chosen_anime_name}:\n") print(f"Recommendations for {chosen_anime_name}:\n")
else: else:
@ -189,56 +177,136 @@ def predict(prediction_model, pivot_table, seed=42, anime="RANDOM", recommendati
def create_model(pivot_table, metric="cosine", algorithm="brute", neighbors=5): def create_model(pivot_table, metric="cosine", algorithm="brute", neighbors=5):
""" """Creates model based on neaarest neighbor for anime prediction."""
Creates model based on neaarest neighbor for anime prediction
"""
pivot_table_matrix = csr_matrix(pivot_table.values) pivot_table_matrix = csr_matrix(pivot_table.values)
model = NearestNeighbors(n_neighbors=neighbors, model = NearestNeighbors(n_neighbors=neighbors, metric=metric, algorithm=algorithm)
metric=metric, algorithm=algorithm)
model.fit(pivot_table_matrix) model.fit(pivot_table_matrix)
return model return model
def handle_arguments(): def handle_arguments():
parser = argparse.ArgumentParser(description='Example script with pyargs') parser = argparse.ArgumentParser(description="Example script with pyargs")
parser.add_argument('--data_limit', '-dl', parser.add_argument(
help='Specify data limit, Recommended at least 500k, set to -1 for no limit', required=False, type=int, default=-1) "--data_limit",
parser.add_argument('--seed', '-s', help='Specify seed', "-dl",
type=int, required=False, default=42) help="Specify data limit, Recommended at least 500k, set to -1 for no limit",
parser.add_argument('--debug', '-d', help='Use debug (more information) prints', required=False,
type=bool, required=False, default=False) type=int,
parser.add_argument('--database', '-db', help='Specify database path', default=-1,
required=False, default="database") )
parser.add_argument(
"--seed", "-s", help="Specify seed", type=int, required=False, default=42
)
parser.add_argument(
"--debug",
"-d",
help="Use debug (more information) prints",
type=bool,
required=False,
default=False,
)
parser.add_argument(
"--database",
"-db",
help="Specify database path",
required=False,
default="database",
)
allowed_metric = ["cosine", "mahalanobis", "euclidean"] allowed_metric = ["cosine", "mahalanobis", "euclidean"]
parser.add_argument('--metric', '-m', help='Specify metric for NearestNeighbor learner', parser.add_argument(
required=False, default="cosine", choices=allowed_metric) "--metric",
allowed_algorithms = ['auto', 'ball_tree', 'kd_tree', 'brute'] "-m",
parser.add_argument('--algorithm', '-a', help='Specify algorithm for Nearest Neighbor learner', help="Specify metric for NearestNeighbor learner",
required=False, default="brute", choices=allowed_algorithms) required=False,
parser.add_argument('--anime', '-an', help='Specify anime to choose', default="cosine",
required=False, default="RANDOM") choices=allowed_metric,
parser.add_argument('--neighbors', '-n', help='Specify number of nearest neighbors', )
required=False, default=5) allowed_algorithms = ["auto", "ball_tree", "kd_tree", "brute"]
parser.add_argument('--user_threshold', '-ut', help='Specify minimal number of votes required for user to be included in the data, set to -1 for no threshold', parser.add_argument(
required=False, type=int, default=500) "--algorithm",
parser.add_argument('--anime_threshold', '-at', help='Specify minimal number of votes required for anime to be included in the data, set to -1 for no threshold', "-a",
required=False, type=int, default=200) help="Specify algorithm for Nearest Neighbor learner",
parser.add_argument('--recommendation_amount', '-ra', help='Specify how much anime should be recommended', required=False,
required=False, type=int, default=5) default="brute",
choices=allowed_algorithms,
)
parser.add_argument(
"--anime",
"-an",
help="Specify anime to choose",
required=False,
default="RANDOM",
)
parser.add_argument(
"--neighbors",
"-n",
help="Specify number of nearest neighbors",
required=False,
default=5,
)
parser.add_argument(
"--user_threshold",
"-ut",
help="Specify minimal number of votes required for user to be included in the data, set to -1 for no threshold",
required=False,
type=int,
default=500,
)
parser.add_argument(
"--anime_threshold",
"-at",
help="Specify minimal number of votes required for anime to be included in the data, set to -1 for no threshold",
required=False,
type=int,
default=200,
)
parser.add_argument(
"--recommendation_amount",
"-ra",
help="Specify how much anime should be recommended",
required=False,
type=int,
default=5,
)
# Parse the command-line arguments # Parse the command-line arguments
args = parser.parse_args() args = parser.parse_args()
args.recommendation_amount = args.recommendation_amount + 1 args.recommendation_amount = args.recommendation_amount + 1
# Access the values of the arguments # Access the values of the arguments
return args.seed, args.debug, args.data_limit, args.database, args.metric, args.algorithm, args.anime, args.neighbors, args.user_threshold, args.anime_threshold, args.recommendation_amount return (
args.seed,
args.debug,
args.data_limit,
args.database,
args.metric,
args.algorithm,
args.anime,
args.neighbors,
args.user_threshold,
args.anime_threshold,
args.recommendation_amount,
)
if __name__ == "__main__": if __name__ == "__main__":
seed, debug, data_limit, db, metric, algorithm, anime, neighbors, user_threshold, anime_threshold, recommendation_amount = handle_arguments() (
seed,
debug,
data_limit,
db,
metric,
algorithm,
anime,
neighbors,
user_threshold,
anime_threshold,
recommendation_amount,
) = handle_arguments()
RATING_DATA, ANIME_CONTACT_DATA = get_data(data_limit, db) RATING_DATA, ANIME_CONTACT_DATA = get_data(data_limit, db)
PIVOT_TABLE = preprocessing( PIVOT_TABLE = preprocessing(
RATING_DATA, ANIME_CONTACT_DATA, debug, user_threshold, anime_threshold) RATING_DATA, ANIME_CONTACT_DATA, debug, user_threshold, anime_threshold
)
MODEL = create_model(PIVOT_TABLE, metric, algorithm, neighbors) MODEL = create_model(PIVOT_TABLE, metric, algorithm, neighbors)
predict(MODEL, PIVOT_TABLE, seed, anime, recommendation_amount) predict(MODEL, PIVOT_TABLE, seed, anime, recommendation_amount)

269
Programming/ECOTE/program/translator/main.py
View File

@ -1,23 +1,19 @@
""" """Program for converting Latex files into html files."""
Program for converting Latex files into html files
"""
import argparse import argparse
def command_name_check(latex_string, command_name): def command_name_check(latex_string, command_name):
""" """Checks if command name given from latex was spelled correctly."""
Checks if command name given from latex was spelled correctly if latex_string[1 : (len(command_name) + 1)] != command_name:
"""
if latex_string[1: (len(command_name) + 1)] != command_name:
print(latex_string + "Error! command misspeled!") print(latex_string + "Error! command misspeled!")
return "Error!" return "Error!"
return "" return ""
def generic_checks(latex_string): def generic_checks(latex_string):
""" """Checks if string is given and whether latex parameters
Checks if string is given and whether latex parameters ends with curly bracket.
ends with curly bracket
""" """
if latex_string == "": if latex_string == "":
print(latex_string + "Error! No input given to function") print(latex_string + "Error! No input given to function")
@ -29,8 +25,7 @@ def generic_checks(latex_string):
def generic_checks_command(latex_string): def generic_checks_command(latex_string):
""" """In additon to generic checks it also checks if command starts with \
In additon to generic checks it also checks if command starts with \
""" """
if generic_checks(latex_string) == "Error!": if generic_checks(latex_string) == "Error!":
return "Error!" return "Error!"
@ -41,9 +36,7 @@ def generic_checks_command(latex_string):
def document_class_only_checks(latex_string): def document_class_only_checks(latex_string):
""" """Checks only errors connected to document class."""
Checks only errors connected to document class
"""
if len(latex_string) < len("\\documentclass{"): if len(latex_string) < len("\\documentclass{"):
print(latex_string + "Error! latex string to short to be document class!") print(latex_string + "Error! latex string to short to be document class!")
return "Error!" return "Error!"
@ -56,9 +49,8 @@ def document_class_only_checks(latex_string):
def document_class(latex_string): def document_class(latex_string):
r""" r"""Converts LaTeX documentclass method to html
Converts LaTeX documentclass method to html \documentclass{article}.
\documentclass{article}
""" """
if generic_checks_command(latex_string) == "Error!": if generic_checks_command(latex_string) == "Error!":
return "Error!" return "Error!"
@ -70,9 +62,8 @@ def document_class(latex_string):
def begin_document(latex_string): def begin_document(latex_string):
r""" r"""Converts LaTeX begin document method to html
Converts LaTeX begin document method to html \begin{document}.
\begin{document}
""" """
if generic_checks_command(latex_string) == "Error!": if generic_checks_command(latex_string) == "Error!":
return "Error!" return "Error!"
@ -82,9 +73,7 @@ def begin_document(latex_string):
def begin_tabular(latex_string): def begin_tabular(latex_string):
r""" r"""Checks if LaTeX begin tabular method is correct."""
Checks if LaTeX begin tabular method is correct
"""
if generic_checks_command(latex_string) == "Error!": if generic_checks_command(latex_string) == "Error!":
return "Error!" return "Error!"
if command_name_check(latex_string, "begin") == "Error!": if command_name_check(latex_string, "begin") == "Error!":
@ -93,9 +82,7 @@ def begin_tabular(latex_string):
def tabular_parameters(latex_string): def tabular_parameters(latex_string):
r""" r"""Checks if LaTeX tabular environment has any parameters."""
Checks if LaTeX tabular environment has any parameters
"""
if latex_string == "": if latex_string == "":
return "" return ""
if latex_string == "[]": if latex_string == "[]":
@ -105,16 +92,15 @@ def tabular_parameters(latex_string):
def only_pipes_and_space(latex_string): def only_pipes_and_space(latex_string):
"""Checks if latex string only contains | or " " """Checks if latex string only contains | or " "
if yes returns True, if no returns false""" if yes returns True, if no returns false.
"""
return all(char in ("|", " ", "{", "}") for char in latex_string) return all(char in ("|", " ", "{", "}") for char in latex_string)
def handle_complicated_parameter(latex_string, return_array, i): def handle_complicated_parameter(latex_string, return_array, i):
""" """Handles single complicated parameter from loop."""
handles single complicated parameter from loop
"""
closing_bracket = latex_string.find("}", i + 1) closing_bracket = latex_string.find("}", i + 1)
columns_string = latex_string[i: closing_bracket + 1] columns_string = latex_string[i : closing_bracket + 1]
result = tabular_columns_parameters(columns_string) result = tabular_columns_parameters(columns_string)
if result == "Error!": if result == "Error!":
return return_array, i, result return return_array, i, result
@ -123,16 +109,13 @@ def handle_complicated_parameter(latex_string, return_array, i):
return return_array, i, "" return return_array, i, ""
def main_tabular_parameters_loop_inside(latex_string, i, def main_tabular_parameters_loop_inside(
simple_parameters_dictionary, latex_string, i, simple_parameters_dictionary, return_array
return_array): ):
""" """Handles single tabular parameter character."""
Handles single tabular parameter character
"""
character = latex_string[i] character = latex_string[i]
if character in ["l", "c", "r", "|"]: if character in ["l", "c", "r", "|"]:
return_array.append( return_array.append(simple_parameters_dictionary.get(latex_string[i]))
simple_parameters_dictionary.get(latex_string[i]))
i += 1 i += 1
return return_array, i, "" return return_array, i, ""
if character in ["p", "m", "b"]: if character in ["p", "m", "b"]:
@ -142,24 +125,22 @@ def main_tabular_parameters_loop_inside(latex_string, i,
def main_tabular_parameters_loop(latex_string, simple_parameters_dictionary): def main_tabular_parameters_loop(latex_string, simple_parameters_dictionary):
""" """Converts simple and paragraph table parameters into html style code."""
Converts simple and paragraph table parameters into html style code
"""
return_array = [] return_array = []
i = 0 i = 0
latex_string_length = len(latex_string) latex_string_length = len(latex_string)
while i < latex_string_length: while i < latex_string_length:
return_array, i, result = main_tabular_parameters_loop_inside( return_array, i, result = main_tabular_parameters_loop_inside(
latex_string, i, simple_parameters_dictionary, return_array) latex_string, i, simple_parameters_dictionary, return_array
)
if result == "Error!": if result == "Error!":
return result return result
return return_array return return_array
def tabular_required_errors(latex_string): def tabular_required_errors(latex_string):
""" """Runs generic checks and
Runs generic checks and checks if there are any parameters put to latex table.
checks if there are any parameters put to latex table
""" """
if generic_checks(latex_string) == "Error!": if generic_checks(latex_string) == "Error!":
return "Error!" return "Error!"
@ -174,9 +155,7 @@ def tabular_required_errors(latex_string):
def simple_parameters_dictionary_return(): def simple_parameters_dictionary_return():
""" """Defines html styles for simple tabular parameters."""
Defines html styles for simple tabular parameters
"""
simple_parameters_dictionary = { simple_parameters_dictionary = {
"l": "align='left'", "l": "align='left'",
"c": "align='center'", "c": "align='center'",
@ -187,9 +166,7 @@ def simple_parameters_dictionary_return():
def tabular_required_parameters(latex_string): def tabular_required_parameters(latex_string):
""" """Converts simple tabular parameters to html style commands."""
Converts simple tabular parameters to html style commands
"""
if tabular_required_errors(latex_string) == "Error!": if tabular_required_errors(latex_string) == "Error!":
return "Error!" return "Error!"
simple_parameters_dictionary = simple_parameters_dictionary_return() simple_parameters_dictionary = simple_parameters_dictionary_return()
@ -200,14 +177,11 @@ def tabular_required_parameters(latex_string):
latex_string, latex_string,
) )
return "Error!" return "Error!"
return main_tabular_parameters_loop( return main_tabular_parameters_loop(latex_string, simple_parameters_dictionary)
latex_string, simple_parameters_dictionary)
def length_conversions(latex_length): def length_conversions(latex_length):
""" """Converts latex lengths to html lengths."""
Converts latex lengths to html lengths
"""
length_dictionary = { length_dictionary = {
"pt": [1.3, "px"], "pt": [1.3, "px"],
"mm": [1, "mm"], "mm": [1, "mm"],
@ -220,9 +194,7 @@ def length_conversions(latex_length):
def construct_return_string(vertical_align_type, final_length, conversed_unit): def construct_return_string(vertical_align_type, final_length, conversed_unit):
""" """Construsts entire html style of parameters."""
Construsts entire html style of parameters
"""
return_string = ( return_string = (
"style='" "style='"
+ vertical_align_type + vertical_align_type
@ -235,9 +207,7 @@ def construct_return_string(vertical_align_type, final_length, conversed_unit):
def return_parameter_dictionary(): def return_parameter_dictionary():
""" """Returns html styles for each complicated latex parameter."""
Returns html styles for each complicated latex parameter
"""
parameter_dictionary = { parameter_dictionary = {
"p": "vertical-align: top;", "p": "vertical-align: top;",
"m": "vertical-align: middle;", "m": "vertical-align: middle;",
@ -247,13 +217,11 @@ def return_parameter_dictionary():
def tabular_columns_parameters_errors(latex_string, vertical_align_type): def tabular_columns_parameters_errors(latex_string, vertical_align_type):
""" """Checks for intermediate errors
Checks for intermediate errors Checks if parameter length starts with curly bracket.
Checks if parameter length starts with curly bracket
""" """
if vertical_align_type == "Error!": if vertical_align_type == "Error!":
print("tabular_columns_parameters, unknown parameter: ", print("tabular_columns_parameters, unknown parameter: ", latex_string[0])
latex_string[0])
return "Error!" return "Error!"
if latex_string[1] != "{": if latex_string[1] != "{":
print( print(
@ -266,9 +234,7 @@ def tabular_columns_parameters_errors(latex_string, vertical_align_type):
def string_length_no_whitespace(length_parameter): def string_length_no_whitespace(length_parameter):
""" """Calculates string width without whitespace."""
Calculates string width without whitespace
"""
i = 0 i = 0
length_value = "" length_value = ""
for character in length_parameter: for character in length_parameter:
@ -280,15 +246,12 @@ def string_length_no_whitespace(length_parameter):
def tabular_columns_parameters(latex_string): def tabular_columns_parameters(latex_string):
""" """Handles tabular paragraph parameters and translates them to html style."""
Handles tabular paragraph parameters and translates them to html style
"""
if generic_checks(latex_string) == "Error!": if generic_checks(latex_string) == "Error!":
return "Error!" return "Error!"
parameter_dictionary = return_parameter_dictionary() parameter_dictionary = return_parameter_dictionary()
vertical_align_type = parameter_dictionary.get(latex_string[0], "Error!") vertical_align_type = parameter_dictionary.get(latex_string[0], "Error!")
if tabular_columns_parameters_errors( if tabular_columns_parameters_errors(latex_string, vertical_align_type) == "Error!":
latex_string, vertical_align_type) == "Error!":
return "Error!" return "Error!"
length_parameter_with_bracket = latex_string.partition("{")[2] length_parameter_with_bracket = latex_string.partition("{")[2]
length_parameter = length_parameter_with_bracket.partition("}")[0] length_parameter = length_parameter_with_bracket.partition("}")[0]
@ -296,50 +259,44 @@ def tabular_columns_parameters(latex_string):
length_unit = length_parameter[i:] length_unit = length_parameter[i:]
conversed_unit = length_conversions(length_unit) conversed_unit = length_conversions(length_unit)
if conversed_unit == "Error!": if conversed_unit == "Error!":
print("""tabular_columns_parameters, print(
Unit could not be conversed!""", latex_string) """tabular_columns_parameters,
Unit could not be conversed!""",
latex_string,
)
return "Error!" return "Error!"
final_length = round(float(length_value) * conversed_unit[0], 2) final_length = round(float(length_value) * conversed_unit[0], 2)
return construct_return_string(vertical_align_type, return construct_return_string(vertical_align_type, final_length, conversed_unit)
final_length, conversed_unit)
def split_rows(latex_string): def split_rows(latex_string):
""" """Splits table row string into separate strings and puts them in array."""
Splits table row string into separate strings and puts them in array
"""
double_backslash = "\\" double_backslash = "\\"
rows = latex_string.split(double_backslash) rows = latex_string.split(double_backslash)
return rows return rows
def split_columns(table_row, column_count): def split_columns(table_row, column_count):
""" """Splits table row columns into separate strings and puts them in array."""
Splits table row columns into separate strings and puts them in array
"""
columns = table_row.split("&") columns = table_row.split("&")
if len(columns) != column_count and columns != [""]: if len(columns) != column_count and columns != [""]:
print( print(
f"Error! extracted columns length: {columns} and columns_count: {column_count} is different!") f"Error! extracted columns length: {columns} and columns_count: {column_count} is different!"
)
return "Error!" return "Error!"
return columns return columns
def translate_column(latex_column): def translate_column(latex_column):
""" """Translate insides of a single latex tabular column to html."""
Translate insides of a single latex tabular column to html
"""
hline_string_literal = "\\hline" hline_string_literal = "\\hline"
replaced_hline = latex_column.replace(hline_string_literal, "<hr>") replaced_hline = latex_column.replace(hline_string_literal, "<hr>")
replaced_newline = replaced_hline.replace("\newline", "<br>") replaced_newline = replaced_hline.replace("\newline", "<br>")
return replaced_newline return replaced_newline
def handle_line_strings(line_string, def handle_line_strings(line_string, column_style, column_number, return_string):
column_style, column_number, return_string): """Converts lines untill there are no more lines."""
"""
Converts lines untill there are no more lines
"""
current_style = column_style[column_number] current_style = column_style[column_number]
if current_style == line_string: if current_style == line_string:
while current_style == line_string: while current_style == line_string:
@ -349,11 +306,10 @@ def handle_line_strings(line_string,
return return_string, column_number return return_string, column_number
def handle_single_column(column, column_style, def handle_single_column(
column_number, line_string, return_string): column, column_style, column_number, line_string, return_string
""" ):
Handles single column from columns loop """Handles single column from columns loop."""
"""
return_string += "<td " return_string += "<td "
if column_number >= len(column_style): if column_number >= len(column_style):
print( print(
@ -362,7 +318,8 @@ def handle_single_column(column, column_style,
) )
return "Error!", column_number return "Error!", column_number
return_string, column_number = handle_line_strings( return_string, column_number = handle_line_strings(
line_string, column_style, column_number, return_string) line_string, column_style, column_number, return_string
)
return_string += column_style[column_number] + ">" return_string += column_style[column_number] + ">"
return_string += translate_column(column) return_string += translate_column(column)
@ -371,42 +328,36 @@ def handle_single_column(column, column_style,
return return_string, column_number return return_string, column_number
def handle_latex_columns(columns, column_style, def handle_latex_columns(
column_number, line_string, return_string): columns, column_style, column_number, line_string, return_string
""" ):
Goes through every column in a row and translates it to html """Goes through every column in a row and translates it to html."""
"""
if columns != "Error!": if columns != "Error!":
for column in columns: for column in columns:
return_string, column_number = handle_single_column(column, return_string, column_number = handle_single_column(
column_style, column, column_style, column_number, line_string, return_string
column_number, )
line_string,
return_string)
return return_string, column_number return return_string, column_number
def go_through_latex_rows(rows, return_string, def go_through_latex_rows(
column_amount, column_style, line_string): rows, return_string, column_amount, column_style, line_string
""" ):
Goes through latex table insides row by row """Goes through latex table insides row by row."""
"""
for row in rows: for row in rows:
return_string += "<tr>" return_string += "<tr>"
columns = split_columns(row, column_amount) columns = split_columns(row, column_amount)
column_number = 0 column_number = 0
return_string, column_number = handle_latex_columns( return_string, column_number = handle_latex_columns(
columns, column_style, column_number, line_string, return_string) columns, column_style, column_number, line_string, return_string
)
return_string += "</tr>" return_string += "</tr>"
return return_string return return_string
def translate_inside_to_html(latex_table_inside, column_style): def translate_inside_to_html(latex_table_inside, column_style):
""" """Translates entire table insides to html."""
Translates entire table insides to html
"""
return_string = "<table>" return_string = "<table>"
column_amount = 0 column_amount = 0
line_string = "style='border-left: 1px solid black'" line_string = "style='border-left: 1px solid black'"
@ -415,65 +366,58 @@ def translate_inside_to_html(latex_table_inside, column_style):
column_amount += 1 column_amount += 1
rows = split_rows(latex_table_inside) rows = split_rows(latex_table_inside)
return_string = go_through_latex_rows( return_string = go_through_latex_rows(
rows, return_string, column_amount, column_style, line_string) rows, return_string, column_amount, column_style, line_string
)
return_string += "</table>" return_string += "</table>"
return return_string return return_string
def read_file(tex_filename): def read_file(tex_filename):
"""opens, reads and saves latex file
content into python string, then closes file.
""" """
opens, reads and saves latex file with open(tex_filename, encoding="UTF-8") as tex_file:
content into python string, then closes file
"""
with open(tex_filename, "r", encoding="UTF-8") as tex_file:
data = tex_file.read().replace("\n", "") data = tex_file.read().replace("\n", "")
tex_file.close() tex_file.close()
return data return data
def read_document_class(latex_full_string): def read_document_class(latex_full_string):
""" """Finds documentclass in latex file and checks function correctness."""
Finds documentclass in latex file and checks function correctness
"""
document_class_index = latex_full_string.find("\\documentclass") document_class_index = latex_full_string.find("\\documentclass")
if document_class_index == -1: if document_class_index == -1:
print("Main function error! documentclass not found") print("Main function error! documentclass not found")
return "Error!" return "Error!"
document_class_close_bracket = latex_full_string.find( document_class_close_bracket = latex_full_string.find("}", document_class_index)
"}", document_class_index)
document_class_string = latex_full_string[ document_class_string = latex_full_string[
document_class_index: document_class_close_bracket + 1 document_class_index : document_class_close_bracket + 1
] ]
latex_full_string = latex_full_string[ latex_full_string = latex_full_string[
document_class_close_bracket + 1: len(latex_full_string) - 1 document_class_close_bracket + 1 : len(latex_full_string) - 1
] ]
return document_class(document_class_string), latex_full_string return document_class(document_class_string), latex_full_string
def read_begin_document(latex_full_string): def read_begin_document(latex_full_string):
""" """Finds LaTeX \begin{document} command and returns html counterpart to it."""
Finds LaTeX \begin{document} command and returns html counterpart to it
"""
begin_document_index = latex_full_string.find(r"\begin{document}") begin_document_index = latex_full_string.find(r"\begin{document}")
if begin_document_index == -1: if begin_document_index == -1:
print("read_begin_document error! begin{document not found") print("read_begin_document error! begin{document not found")
return "Error!", latex_full_string return "Error!", latex_full_string
begin_document_close_bracket = latex_full_string.find( begin_document_close_bracket = latex_full_string.find("}", begin_document_index)
"}", begin_document_index)
begin_document_string = latex_full_string[ begin_document_string = latex_full_string[
begin_document_index: begin_document_close_bracket + 1 begin_document_index : begin_document_close_bracket + 1
] ]
latex_full_string = latex_full_string[ latex_full_string = latex_full_string[
begin_document_close_bracket + 1: len(latex_full_string) - 1 begin_document_close_bracket + 1 : len(latex_full_string) - 1
] ]
return_string = begin_document(begin_document_string) return_string = begin_document(begin_document_string)
return return_string, latex_full_string return return_string, latex_full_string
def read_start(html_string, data): def read_start(html_string, data):
""" """Finds, reads latex document class
Finds, reads latex document class and begin document and translates them to html.
and begin document and translates them to html
""" """
document_class_result, data = read_document_class(data) document_class_result, data = read_document_class(data)
if document_class_result == "Error!": if document_class_result == "Error!":
@ -488,28 +432,24 @@ def read_start(html_string, data):
def handle_table_whole(html_string, data, table_start, table_end): def handle_table_whole(html_string, data, table_start, table_end):
""" """Reads entire table, handles its parameters
Reads entire table, handles its parameters and insides and translates to html.
and insides and translates to html
""" """
if table_start == -1 or table_end == -1: if table_start == -1 or table_end == -1:
return html_string, data return html_string, data
tabular_begin_string = r"\begin{tabular}" tabular_begin_string = r"\begin{tabular}"
parameters_start_index = data.find( parameters_start_index = data.find("{", table_start + len(tabular_begin_string))
"{", table_start + len(tabular_begin_string))
parameters_end_index = data.find("}", parameters_start_index) parameters_end_index = data.find("}", parameters_start_index)
parameters_string = data[parameters_start_index: parameters_end_index + 1] parameters_string = data[parameters_start_index : parameters_end_index + 1]
parameters_array = tabular_required_parameters(parameters_string) parameters_array = tabular_required_parameters(parameters_string)
inside_table = data[parameters_end_index + 1: table_end] inside_table = data[parameters_end_index + 1 : table_end]
html_string += translate_inside_to_html(inside_table, parameters_array) html_string += translate_inside_to_html(inside_table, parameters_array)
return html_string return html_string
def handle_insides(html_string, data): def handle_insides(html_string, data):
""" """Finds inside of table and translates them to html."""
Finds inside of table and translates them to html
"""
tabular_index = 0 tabular_index = 0
tabular_end_string = r"\end{tabular}" tabular_end_string = r"\end{tabular}"
while tabular_index != -1: while tabular_index != -1:
@ -522,15 +462,14 @@ def handle_insides(html_string, data):
html_string = handle_table_whole( html_string = handle_table_whole(
html_string, data, tabular_index_start, tabular_index_end html_string, data, tabular_index_start, tabular_index_end
) )
data = data[tabular_index_end + len(tabular_end_string): len(data)] data = data[tabular_index_end + len(tabular_end_string) : len(data)]
tabular_index = tabular_index_end tabular_index = tabular_index_end
return html_string, data return html_string, data
def main_function(tex_filename): def main_function(tex_filename):
""" """Ran by default by program,
Ran by default by program, gets latex file and translates to html.
gets latex file and translates to html
""" """
data = read_file(tex_filename) data = read_file(tex_filename)
html_string = "" html_string = ""
@ -540,15 +479,19 @@ def main_function(tex_filename):
when reading documentclass or begin document""") when reading documentclass or begin document""")
return "Error" return "Error"
html_string, data = handle_insides(html_string, data) html_string, data = handle_insides(html_string, data)
html_string = html_string.replace('}', '') html_string = html_string.replace("}", "")
return html_string return html_string
def handle_arguments(): def handle_arguments():
parser = argparse.ArgumentParser( parser = argparse.ArgumentParser(description="Specify tex filename and path")
description='Specify tex filename and path') parser.add_argument(
parser.add_argument('--filename', '-f', "--filename",
help='Specify tex filename, include path if needed ', required=False, default="texfile.tex") "-f",
help="Specify tex filename, include path if needed ",
required=False,
default="texfile.tex",
)
args = parser.parse_args() args = parser.parse_args()
return args.filename return args.filename

36
Programming/ECOTE/program/unit_tests/test_code/test_begin_document.py
View File

@ -1,59 +1,53 @@
""" """Tests begin document function."""
Tests begin document function
"""
from translator.main import begin_document from translator.main import begin_document
# Write python tests for a function translating LaTeX documentclass to html # Write python tests for a function translating LaTeX documentclass to html
def given_empty_then_error(): def given_empty_then_error():
""" """Given: ""
Given: ""
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert begin_document("") == "Error!" assert begin_document("") == "Error!"
def given_not_closed_then_error(): def given_not_closed_then_error():
r""" r"""Given: "\\begin\{document"
Given: "\\begin\{document"
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert begin_document(r"\\begin{document") == "Error!" assert begin_document(r"\\begin{document") == "Error!"
def given_no_opening_then_error(): def given_no_opening_then_error():
""" """Given: No opening curly bracket
Given: No opening curly bracket
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert begin_document(r"\\begindocument}") == "Error!" assert begin_document(r"\\begindocument}") == "Error!"
def given_misspeled_then_error(): def given_misspeled_then_error():
""" """Given: misspelled begin document
Given: misspelled begin document
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert begin_document(r"\\begim{document}") == "Error!" assert begin_document(r"\\begim{document}") == "Error!"
def given_no_slash_then_error(): def given_no_slash_then_error():
""" """Given: no backslash at start
Given: no backslash at start
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert begin_document(r"begin{document}") == "Error!" assert begin_document(r"begin{document}") == "Error!"
def given_correct_then_html(): def given_correct_then_html():
""" """Given: \\documentclass{article}
Given: \\documentclass{article}
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert begin_document(r"\begin{document}") == "<html>" assert begin_document(r"\begin{document}") == "<html>"

42
Programming/ECOTE/program/unit_tests/test_code/test_begin_tabular.py
View File

@ -1,67 +1,61 @@
""" """Tests begin document function."""
Tests begin document function
"""
from translator.main import begin_tabular from translator.main import begin_tabular
# Write python tests for a function translating LaTeX documentclass to html # Write python tests for a function translating LaTeX documentclass to html
def given_empty_then_error(): def given_empty_then_error():
""" """Given: ""
Given: ""
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert begin_tabular("") == "Error!" assert begin_tabular("") == "Error!"
def given_not_closed_then_error(): def given_not_closed_then_error():
r""" r"""Given: command not closed
Given: command not closed
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert begin_tabular(r"\\begin{tabular") == "Error!" assert begin_tabular(r"\\begin{tabular") == "Error!"
def given_no_opening_then_error(): def given_no_opening_then_error():
""" """Given: No opening curly bracket
Given: No opening curly bracket
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert begin_tabular(r"\\begintabular}") == "Error!" assert begin_tabular(r"\\begintabular}") == "Error!"
def given_misspeled_then_error(): def given_misspeled_then_error():
""" """Given: misspelled begin tabular
Given: misspelled begin tabular
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert begin_tabular(r"\\begim{tabular}") == "Error!" assert begin_tabular(r"\\begim{tabular}") == "Error!"
def given_no_slash_then_error(): def given_no_slash_then_error():
""" """Given: no backslash at start
Given: no backslash at start
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert begin_tabular(r"begin{tabular}") == "Error!" assert begin_tabular(r"begin{tabular}") == "Error!"
def given_tabular_star_then_error(): def given_tabular_star_then_error():
""" """Given: no backslash at start
Given: no backslash at start
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert begin_tabular(r"begin{tabular*}") == "Error!" assert begin_tabular(r"begin{tabular*}") == "Error!"
def given_correct_then_html(): def given_correct_then_html():
""" """Given: \begin{tabular}
Given: \begin{tabular}
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert begin_tabular(r"\begin{tabular}") == "<table>" assert begin_tabular(r"\begin{tabular}") == "<table>"

50
Programming/ECOTE/program/unit_tests/test_code/test_document_class.py
View File

@ -1,82 +1,70 @@
""" """Tests document class function."""
Tests document class function
"""
from translator.main import document_class from translator.main import document_class
# Write python tests for a function translating LaTeX documentclass to html # Write python tests for a function translating LaTeX documentclass to html
def given_empty_then_error(): def given_empty_then_error():
""" """Given: ""
Given: ""
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert document_class("") == "Error!" assert document_class("") == "Error!"
def given_not_closed_then_error(): def given_not_closed_then_error():
""" """Given: "\\documentclass{article"
Given: "\\documentclass{article"
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert document_class("\\documentclass{article") == "Error!" assert document_class("\\documentclass{article") == "Error!"
def given_parameters_then_error(): def given_parameters_then_error():
""" """Given: class parameters
Given: class parameters
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert document_class("\\documentclass[12pt]{article}") == "Error!" assert document_class("\\documentclass[12pt]{article}") == "Error!"
def given_no_opening_then_error(): def given_no_opening_then_error():
""" """Given: No opening curly bracket
Given: No opening curly bracket
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert document_class("\\documentclassarticle}") == "Error!" assert document_class("\\documentclassarticle}") == "Error!"
def given_misspeled_then_error(): def given_misspeled_then_error():
""" """Given: misspelled document class
Given: misspelled document class
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert document_class("\\documentclasZ{article}") == "Error!" assert document_class("\\documentclasZ{article}") == "Error!"
def given_class_not_recognized_then_error(): def given_class_not_recognized_then_error():
""" """Given: class not recognized
Given: class not recognized
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert ( assert document_class("\\documentclass{idonotexist}") == "<!DOCTYPE html>"
document_class("\\documentclass{idonotexist}")
== "<!DOCTYPE html>"
)
def given_no_slash_then_error(): def given_no_slash_then_error():
""" """Given: no backslash at start
Given: no backslash at start
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert document_class("documentclass{article}") == "Error!" assert document_class("documentclass{article}") == "Error!"
def given_correct_then_html(): def given_correct_then_html():
""" """Given: \\documentclass{article}
Given: \\documentclass{article}
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert document_class("\\documentclass{article}") == "<!DOCTYPE html>" assert document_class("\\documentclass{article}") == "<!DOCTYPE html>"

6
Programming/ECOTE/program/unit_tests/test_code/test_generic_checks.py
View File

@ -6,13 +6,11 @@ def test_empty_string():
def test_no_curly_bracket(): def test_no_curly_bracket():
assert generic_checks( assert generic_checks("latex_string_without_curly_bracket") == "Error!"
"latex_string_without_curly_bracket") == "Error!"
def test_with_curly_bracket_at_end(): def test_with_curly_bracket_at_end():
assert generic_checks( assert generic_checks("latex_string_with_curly_bracket}") == ""
"latex_string_with_curly_bracket}") == ""
def test_generic_checks(): def test_generic_checks():

13
Programming/ECOTE/program/unit_tests/test_code/test_generic_checks_command.py
View File

@ -6,18 +6,19 @@ def test_empty_string():
def test_no_curly_bracket(): def test_no_curly_bracket():
assert (generic_checks_command( assert (generic_checks_command("latex_string_without_curly_bracket")) == "Error!"
"latex_string_without_curly_bracket")) == "Error!"
def test_no_slash_at_beginning(): def test_no_slash_at_beginning():
assert (generic_checks_command( assert (
"latex_string_without_slash_at_beginning}")) == "Error!" generic_checks_command("latex_string_without_slash_at_beginning}")
) == "Error!"
def test_with_slash_and_curly_bracket(): def test_with_slash_and_curly_bracket():
assert (generic_checks_command( assert (
"\\latex_string_with_slash_and_curly_bracket}")) == "" generic_checks_command("\\latex_string_with_slash_and_curly_bracket}")
) == ""
def test_generic_checks_command(): def test_generic_checks_command():

55
Programming/ECOTE/program/unit_tests/test_code/test_length_conversions.py
View File

@ -1,97 +1,86 @@
""" """Tests begin document function."""
Tests begin document function
"""
from translator.main import length_conversions from translator.main import length_conversions
# Write python tests for a function translating LaTeX documentclass to html # Write python tests for a function translating LaTeX documentclass to html
def given_empty_then_error(): def given_empty_then_error():
""" """Given: ""
Given: ""
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert length_conversions("") == "Error!" assert length_conversions("") == "Error!"
def given_pt_then_px(): def given_pt_then_px():
""" """Given: ""
Given: ""
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert length_conversions("pt") == [1.3, "px"] assert length_conversions("pt") == [1.3, "px"]
def given_mm_then_mm(): def given_mm_then_mm():
""" """Given: ""
Given: ""
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert length_conversions("mm") == [1, "mm"] assert length_conversions("mm") == [1, "mm"]
def given_cm_then_cm(): def given_cm_then_cm():
""" """Given: ""
Given: ""
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert length_conversions("cm") == [1, "cm"] assert length_conversions("cm") == [1, "cm"]
def given_in_then_in(): def given_in_then_in():
""" """Given: ""
Given: ""
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert length_conversions("in") == [1, "in"] assert length_conversions("in") == [1, "in"]
def given_ex_then_ex(): def given_ex_then_ex():
""" """Given: ""
Given: ""
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert length_conversions("ex") == [1, "ex"] assert length_conversions("ex") == [1, "ex"]
def given_em_then_em(): def given_em_then_em():
""" """Given: ""
Given: ""
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert length_conversions("em") == [1, "em"] assert length_conversions("em") == [1, "em"]
def given_mu_then_error(): def given_mu_then_error():
""" """Given: ""
Given: ""
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert length_conversions("mu") == "Error!" assert length_conversions("mu") == "Error!"
def given_sp_then_error(): def given_sp_then_error():
""" """Given: ""
Given: ""
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert length_conversions("sp") == "Error!" assert length_conversions("sp") == "Error!"
def given_unknown_then_error(): def given_unknown_then_error():
""" """Given: ""
Given: ""
When: N/A When: N/A
Then: Error message Then: Error message.
""" """
assert length_conversions("unknown") == "Error!" assert length_conversions("unknown") == "Error!"

6
Programming/ECOTE/program/unit_tests/test_code/test_split_columns.py
View File

@ -2,15 +2,15 @@ from translator.main import split_columns
def given_empty_then_correct(): def given_empty_then_correct():
assert split_columns("", 0) == [''] assert split_columns("", 0) == [""]
def given_no_and_then_correct(): def given_no_and_then_correct():
assert split_columns("estsegsegtswegseg", 1) == ['estsegsegtswegseg'] assert split_columns("estsegsegtswegseg", 1) == ["estsegsegtswegseg"]
def given_single_and_then_correct(): def given_single_and_then_correct():
assert split_columns("&", 2) == ['', ''] assert split_columns("&", 2) == ["", ""]
def given_too_much_columns_then_error(): def given_too_much_columns_then_error():

4
Programming/ECOTE/program/unit_tests/test_code/test_split_rows.py
View File

@ -2,7 +2,7 @@ from translator.main import split_rows
def given_empty_then_correct(): def given_empty_then_correct():
assert split_rows("") == [''] assert split_rows("") == [""]
def given_no_slash_then_correct(): def given_no_slash_then_correct():
@ -10,7 +10,7 @@ def given_no_slash_then_correct():
def given_double_slash_then_correct(): def given_double_slash_then_correct():
assert split_rows("\\") == ['', ''] assert split_rows("\\") == ["", ""]
def given_actual_string_then_correct(): def given_actual_string_then_correct():

43
Programming/ECOTE/program/unit_tests/test_code/test_tabular_columns_parameters.py
View File

@ -2,51 +2,52 @@ from translator.main import tabular_columns_parameters
def given_empty_then_error(): def given_empty_then_error():
""" """Given:
Given:
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_columns_parameters("") == "Error!" assert tabular_columns_parameters("") == "Error!"
def given_unknown_then_error(): def given_unknown_then_error():
""" """Given:
Given:
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_columns_parameters("l{1.3cm}") == "Error!" assert tabular_columns_parameters("l{1.3cm}") == "Error!"
def given_p_then_array(): def given_p_then_array():
""" """Given:
Given:
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_columns_parameters( assert (
"p{1.3pt}") == "style='vertical-align: top; width: 1.69px;'" tabular_columns_parameters("p{1.3pt}")
== "style='vertical-align: top; width: 1.69px;'"
)
def given_m_then_array(): def given_m_then_array():
""" """Given:
Given:
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_columns_parameters( assert (
"m{1.3pt}") == "style='vertical-align: middle; width: 1.69px;'" tabular_columns_parameters("m{1.3pt}")
== "style='vertical-align: middle; width: 1.69px;'"
)
def given_b_then_array(): def given_b_then_array():
""" """Given:
Given:
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_columns_parameters( assert (
"b{1.3pt}") == "style='vertical-align: bottom; width: 1.69px;'" tabular_columns_parameters("b{1.3pt}")
== "style='vertical-align: bottom; width: 1.69px;'"
)
def test_tabular_columns_parameters(): def test_tabular_columns_parameters():

23
Programming/ECOTE/program/unit_tests/test_code/test_tabular_parameters.py
View File

@ -1,35 +1,32 @@
""" """Tests tabular_parameters function."""
Tests tabular_parameters function
"""
from translator.main import tabular_parameters from translator.main import tabular_parameters
def given_empty_then_empty(): def given_empty_then_empty():
""" """Given:
Given:
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_parameters("") == "" assert tabular_parameters("") == ""
def given_empty_brackets_then_empty(): def given_empty_brackets_then_empty():
""" """Given: []
Given: []
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_parameters("[]") == "" assert tabular_parameters("[]") == ""
def given_non_empty_then_error(): def given_non_empty_then_error():
""" """Given: [c]
Given: [c]
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_parameters("[c]") == "Error!" assert tabular_parameters("[c]") == "Error!"
def test_tabular_parameters(): def test_tabular_parameters():
given_empty_then_empty() given_empty_then_empty()
given_empty_brackets_then_empty() given_empty_brackets_then_empty()

79
Programming/ECOTE/program/unit_tests/test_code/test_tabular_required_parameters.py
View File

@ -1,135 +1,120 @@
""" """Tests tabular_parameters function."""
Tests tabular_parameters function
"""
from translator.main import tabular_required_parameters from translator.main import tabular_required_parameters
def given_empty_then_error(): def given_empty_then_error():
""" """Given:
Given:
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_required_parameters("") == "Error!" assert tabular_required_parameters("") == "Error!"
def given_empty_brackets_then_error(): def given_empty_brackets_then_error():
""" """Given: {}
Given: {}
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_required_parameters("{}") == "Error!" assert tabular_required_parameters("{}") == "Error!"
def given_just_line_then_error(): def given_just_line_then_error():
""" """Given: {}
Given: {}
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_required_parameters("{|}") == "Error!" assert tabular_required_parameters("{|}") == "Error!"
def given_just_lines_then_error(): def given_just_lines_then_error():
""" """Given: {}
Given: {}
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_required_parameters("{|||||||}") == "Error!" assert tabular_required_parameters("{|||||||}") == "Error!"
def given_single_left_then_correct(): def given_single_left_then_correct():
""" """Given: {l}
Given: {l}
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_required_parameters("{l}") == ["align='left'"] assert tabular_required_parameters("{l}") == ["align='left'"]
def given_single_center_then_correct(): def given_single_center_then_correct():
""" """Given: {c}
Given: {c}
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_required_parameters("{c}") == ["align='center'"] assert tabular_required_parameters("{c}") == ["align='center'"]
def given_single_right_then_correct(): def given_single_right_then_correct():
""" """Given: {r}
Given: {r}
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_required_parameters("{r}") == ["align='right'"] assert tabular_required_parameters("{r}") == ["align='right'"]
def given_empty_wrap_p_then_error(): def given_empty_wrap_p_then_error():
""" """Given: {r}
Given: {r}
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_required_parameters("{p}") == "Error!" assert tabular_required_parameters("{p}") == "Error!"
def given_empty_wrap_m_then_error(): def given_empty_wrap_m_then_error():
""" """Given: {r}
Given: {r}
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_required_parameters("{m}") == "Error!" assert tabular_required_parameters("{m}") == "Error!"
def given_empty_wrap_b_then_error(): def given_empty_wrap_b_then_error():
""" """Given: {r}
Given: {r}
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_required_parameters("{b}") == "Error!" assert tabular_required_parameters("{b}") == "Error!"
def given_empty_wrap_p_brackets_then_error(): def given_empty_wrap_p_brackets_then_error():
""" """Given: {r}
Given: {r}
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_required_parameters("{p{}}") == "Error!" assert tabular_required_parameters("{p{}}") == "Error!"
def given_empty_wrap_m_brackets_then_error(): def given_empty_wrap_m_brackets_then_error():
""" """Given: {r}
Given: {r}
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_required_parameters("{m{}}") == "Error!" assert tabular_required_parameters("{m{}}") == "Error!"
def given_empty_wrap_b_brackets_then_error(): def given_empty_wrap_b_brackets_then_error():
""" """Given: {r}
Given: {r}
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_required_parameters("{b{}}") == "Error!" assert tabular_required_parameters("{b{}}") == "Error!"
def given_filled_p_brackets_then_correct(): def given_filled_p_brackets_then_correct():
""" """Given: {r}
Given: {r}
When: N/A When: N/A
Then: <!DOCTYPE html><html> Then: <!DOCTYPE html><html>.
""" """
assert tabular_required_parameters("{p{1.3pt}}") == [ assert tabular_required_parameters("{p{1.3pt}}") == [
"style='vertical-align: top; width: 1.69px;'"] "style='vertical-align: top; width: 1.69px;'"
]
def given_complicated_then_correct(): def given_complicated_then_correct():
@ -140,7 +125,7 @@ def given_complicated_then_correct():
"align='center'", "align='center'",
"style='border-left: 1px solid black'", "style='border-left: 1px solid black'",
"align='right'", "align='right'",
"style='border-left: 1px solid black'" "style='border-left: 1px solid black'",
] ]

8
Programming/ECOTE/program/unit_tests/test_code/test_translate_column.py
View File

@ -14,12 +14,14 @@ def given_just_hline_then_correct():
def given_just_newline_then_correct(): def given_just_newline_then_correct():
assert translate_column('\newline') == "<br>" assert translate_column("\newline") == "<br>"
def given_all_then_correct(): def given_all_then_correct():
assert translate_column( assert (
'\\hline \newline hline newline test') == "<hr> <br> hline newline test" translate_column("\\hline \newline hline newline test")
== "<hr> <br> hline newline test"
)
def test_translate_column(): def test_translate_column():

10
Programming/ECOTE/program/unit_tests/test_code/test_translate_inside_to_html.py
View File

@ -1,14 +1,14 @@
from translator.main import translate_inside_to_html from translator.main import tabular_required_parameters, translate_inside_to_html
from translator.main import tabular_required_parameters
def given_correct_then_correct(): def given_correct_then_correct():
latex_string = "test & 2 & test \\ 4 & 5 & 6 \\" latex_string = "test & 2 & test \\ 4 & 5 & 6 \\"
parameters_string = "{ l | c | r }" parameters_string = "{ l | c | r }"
column_styles = tabular_required_parameters(parameters_string) column_styles = tabular_required_parameters(parameters_string)
assert translate_inside_to_html(latex_string, column_styles) == "<table><tr><td align='left'>test </td><td style='border-left: 1px solid black'align='center'> 2 </td><td style='border-left: 1px solid black'align='right'> test </td></tr><tr><td align='left'> 4 </td><td style='border-left: 1px solid black'align='center'> 5 </td><td style='border-left: 1px solid black'align='right'> 6 </td></tr><tr><td align='left'></td></tr></table>" assert (
translate_inside_to_html(latex_string, column_styles)
== "<table><tr><td align='left'>test </td><td style='border-left: 1px solid black'align='center'> 2 </td><td style='border-left: 1px solid black'align='right'> test </td></tr><tr><td align='left'> 4 </td><td style='border-left: 1px solid black'align='center'> 5 </td><td style='border-left: 1px solid black'align='right'> 6 </td></tr><tr><td align='left'></td></tr></table>"
)
def test_translate_inside_to_html(): def test_translate_inside_to_html():

0
Programming/ECRYPT/test1.py
View File

32
Programming/ECRYPT_PROJECT/sieve/conftest.py
View File

@ -8,14 +8,14 @@ def pytest_addoption(parser):
parser.addoption( parser.addoption(
"--primes", "--primes",
action="store", action="store",
default='100', default="100",
help="Upper bound for primes to test." help="Upper bound for primes to test.",
) )
parser.addoption( parser.addoption(
"--composites", "--composites",
action="store", action="store",
default='100', default="100",
help="Upper bound for composites to test." help="Upper bound for composites to test.",
) )
@ -26,16 +26,26 @@ def pytest_configure(config):
def pytest_generate_tests(metafunc): def pytest_generate_tests(metafunc):
df_prime = pd.read_csv("P-100000.csv", header=None) df_prime = pd.read_csv("P-100000.csv", header=None)
df_prime = df_prime[df_prime.iloc[:, 1] < pytest.primes] df_prime = df_prime[df_prime.iloc[:, 1] < pytest.primes]
primes = set(df_prime.iloc[:, 1]) primes = set(df_prime.iloc[:, 1])
df_composite = pd.read_csv("C-100000.csv", header=None, delimiter="=") df_composite = pd.read_csv("C-100000.csv", header=None, delimiter="=")
# interpret first oclumns as numbers # interpret first oclumns as numbers
df_composite = df_composite[df_composite.iloc[:, 0] < pytest.composites] df_composite = df_composite[df_composite.iloc[:, 0] < pytest.composites]
composites = set(df_composite.iloc[:, 0]) composites = set(df_composite.iloc[:, 0])
if metafunc.function.__name__ == 'test_positives': if metafunc.function.__name__ == "test_positives":
metafunc.parametrize('prime', primes) metafunc.parametrize("prime", primes)
metafunc.parametrize('primes_obtained', [sieve_of_eratosthenes(max(primes) + 1),]) metafunc.parametrize(
"primes_obtained",
[
sieve_of_eratosthenes(max(primes) + 1),
],
)
if metafunc.function.__name__ == 'test_composites': if metafunc.function.__name__ == "test_composites":
metafunc.parametrize('composite', composites) metafunc.parametrize("composite", composites)
metafunc.parametrize('primes_obtained', [sieve_of_eratosthenes(max(composites) + 1),]) metafunc.parametrize(
"primes_obtained",
[
sieve_of_eratosthenes(max(composites) + 1),
],
)

16
Programming/ECRYPT_PROJECT/sieve/project.py
View File

@ -1,6 +1,8 @@
'''Python program to print all Primes Smaller than or equal to N using Sieve of Eratosthenes''' """Python program to print all Primes Smaller than or equal to N using Sieve of Eratosthenes."""
import math import math
def sieve_of_eratosthenes(num: int) -> list[int]: def sieve_of_eratosthenes(num: int) -> list[int]:
"""Finds all prime number from 1 to num. """Finds all prime number from 1 to num.
@ -9,25 +11,26 @@ def sieve_of_eratosthenes(num: int) -> list[int]:
Returns: Returns:
list[int]: List of prime numbers list[int]: List of prime numbers
""" """
if num <= 0: if num <= 0:
raise ValueError("Number should not be negative.") raise ValueError("Number should not be negative.")
# boolean list to store if a number is prime or not # boolean list to store if a number is prime or not
prime = [True] * (num+1) prime = [True] * (num + 1)
p = 2 # starting prime number p = 2 # starting prime number
# we will iterate till square root of num because # we will iterate till square root of num because
# if a number is not prime then it will have a factor less than or equal to square root of that number # if a number is not prime then it will have a factor less than or equal to square root of that number
while p <= math.sqrt(num): while p <= math.sqrt(num):
# If prime[p] is not # If prime[p] is not
# changed, then it is a prime # changed, then it is a prime
if prime[p]: if prime[p]:
i = (p << 1) # i = p * 2 i = p << 1 # i = p * 2
# Updating all multiples of p # Updating all multiples of p
while i <= num: while i <= num:
prime[i] = False prime[i] = False
i += p i += p
p += 1 p += 1
return [p for p in range(2, num+1) if prime[p]] return [p for p in range(2, num + 1) if prime[p]]
def print_sieve(num: int) -> None: def print_sieve(num: int) -> None:
@ -35,13 +38,14 @@ def print_sieve(num: int) -> None:
Args: Args:
num (int): Number up to which the prime numbers should be printed num (int): Number up to which the prime numbers should be printed
""" """
for number in sieve_of_eratosthenes(num): for number in sieve_of_eratosthenes(num):
print(number, end=" ") print(number, end=" ")
# Driver code # Driver code
if __name__ == '__main__': if __name__ == "__main__":
num = int(input("Enter a number: ")) num = int(input("Enter a number: "))
print(f"Following are the prime numbers smaller than or equal to {num}") print(f"Following are the prime numbers smaller than or equal to {num}")
print_sieve(num) print_sieve(num)

3
Programming/ECRYPT_PROJECT/sieve/test_main.py
View File

@ -1,7 +1,8 @@
import pandas as pd
import pytest import pytest
from .project import sieve_of_eratosthenes from .project import sieve_of_eratosthenes
def test_positives(prime, primes_obtained): def test_positives(prime, primes_obtained):
assert prime in primes_obtained assert prime in primes_obtained

36
Programming/ERSMS-project/analytics/analytics.py
View File

@ -1,14 +1,11 @@
from flask import Flask, request, jsonify
from flask_caching import Cache
import psycopg2
import pandas
import json
from configparser import ConfigParser from configparser import ConfigParser
from datetime import datetime
import psycopg2
from flask import Flask, jsonify
from flask_caching import Cache
app = Flask(__name__) app = Flask(__name__)
cache = Cache(config={'CACHE_TYPE': 'SimpleCache'}) cache = Cache(config={"CACHE_TYPE": "SimpleCache"})
db_connector = None db_connector = None
conn = None conn = None
@ -33,13 +30,16 @@ def get_movie_ratings(movie_id):
rating_values = [5, 4, 3, 2, 1] rating_values = [5, 4, 3, 2, 1]
for rating in rating_values: for rating in rating_values:
cursor.execute(""" cursor.execute(
"""
SELECT COUNT(*) as count SELECT COUNT(*) as count
FROM ratings FROM ratings
WHERE rating = %s AND movie_ID = %s; WHERE rating = %s AND movie_ID = %s;
""", (rating, movie_id)) """,
(rating, movie_id),
)
result = cursor.fetchone() result = cursor.fetchone()
ratings[f'{rating}_star'] = result[0] ratings[f"{rating}_star"] = result[0]
cursor.close() cursor.close()
@ -62,11 +62,14 @@ def get_number_of_users():
@cache.cached(timeout=50) @cache.cached(timeout=50)
def get_movie_rating_avg(movie_id): def get_movie_rating_avg(movie_id):
cursor = conn.cursor() cursor = conn.cursor()
cursor.execute(""" cursor.execute(
"""
SELECT AVG(rating) as avg_rating SELECT AVG(rating) as avg_rating
FROM ratings FROM ratings
WHERE movie_ID = %s; WHERE movie_ID = %s;
""", (movie_id,)) """,
(movie_id,),
)
res = cursor.fetchall() res = cursor.fetchall()
cursor.close() cursor.close()
@ -78,11 +81,14 @@ def get_movie_rating_avg(movie_id):
@cache.cached(timeout=50) @cache.cached(timeout=50)
def get_user_ratings(user_id): def get_user_ratings(user_id):
cursor = conn.cursor() cursor = conn.cursor()
cursor.execute(""" cursor.execute(
"""
SELECT movie_ID, rating SELECT movie_ID, rating
FROM ratings FROM ratings
WHERE oauth_ID = %s; WHERE oauth_ID = %s;
""", (user_id,)) """,
(user_id,),
)
res = cursor.fetchall() res = cursor.fetchall()
cursor.close() cursor.close()
@ -101,7 +107,7 @@ if __name__ == "__main__":
database=config["postgres"]["database"], database=config["postgres"]["database"],
user=config["postgres"]["user"], user=config["postgres"]["user"],
password=config["postgres"]["password"], password=config["postgres"]["password"],
port=int(config["postgres"]["port"]) port=int(config["postgres"]["port"]),
) )
except Exception: except Exception:

139
Programming/ERSMS-project/backend/app.py
View File

@ -1,24 +1,25 @@
from flask import Flask, request, jsonify import os
import firebase_admin
import requests
from dotenv import load_dotenv
from firebase_admin import auth, credentials
from flask import Flask, jsonify, request
from flask_cors import CORS from flask_cors import CORS
from flask_sqlalchemy import SQLAlchemy from flask_sqlalchemy import SQLAlchemy
import firebase_admin
from firebase_admin import credentials, auth
from dotenv import load_dotenv
import os
import requests
load_dotenv() load_dotenv()
TMDB_BEARER_TOKEN = os.getenv('TMDB_BEARER_TOKEN') TMDB_BEARER_TOKEN = os.getenv("TMDB_BEARER_TOKEN")
app = Flask(__name__) app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///app.sqlite3' app.config["SQLALCHEMY_DATABASE_URI"] = "sqlite:///app.sqlite3"
app.config['SQLALCHEY_TRACK_MODIFICATIONS'] = False app.config["SQLALCHEY_TRACK_MODIFICATIONS"] = False
db = SQLAlchemy(app) db = SQLAlchemy(app)
CORS(app) CORS(app)
cred = credentials.Certificate('movie-recommendation-firebase-adminsdk.json') cred = credentials.Certificate("movie-recommendation-firebase-adminsdk.json")
firebase_admin.initialize_app(cred) firebase_admin.initialize_app(cred)
@ -31,7 +32,7 @@ class User(db.Model):
class Rating(db.Model): class Rating(db.Model):
id = db.Column(db.Integer, primary_key=True) id = db.Column(db.Integer, primary_key=True)
user_id = db.Column(db.Integer, db.ForeignKey('user.uid'), nullable=False) user_id = db.Column(db.Integer, db.ForeignKey("user.uid"), nullable=False)
movie_id = db.Column(db.Integer, nullable=False) movie_id = db.Column(db.Integer, nullable=False)
value = db.Column(db.Integer, nullable=False) value = db.Column(db.Integer, nullable=False)
@ -40,14 +41,14 @@ with app.app_context():
db.create_all() db.create_all()
@app.route('/login', methods=['POST']) @app.route("/login", methods=["POST"])
def login(): def login():
token = request.json.get('token') token = request.json.get("token")
try: try:
decoded_token = auth.verify_id_token(token) decoded_token = auth.verify_id_token(token)
uid = decoded_token['uid'] uid = decoded_token["uid"]
print(uid) print(uid)
email = decoded_token.get('email') email = decoded_token.get("email")
user = User.query.filter_by(uid=uid).first() user = User.query.filter_by(uid=uid).first()
if user is None: if user is None:
@ -56,36 +57,39 @@ def login():
db.session.add(user) db.session.add(user)
db.session.commit() db.session.commit()
return jsonify({'message': 'Login successful!', 'email': email, 'is_admin': user.is_admin}), 200 return jsonify(
{"message": "Login successful!", "email": email, "is_admin": user.is_admin}
), 200
except Exception as e: except Exception as e:
print(e) print(e)
return jsonify({'message': 'Login failed'}), 401 return jsonify({"message": "Login failed"}), 401
@app.route('/count_user_ratings', methods=['POST']) @app.route("/count_user_ratings", methods=["POST"])
def count_user_ratings(): def count_user_ratings():
token = request.json.get('token') token = request.json.get("token")
try: try:
decoded_token = auth.verify_id_token(token) decoded_token = auth.verify_id_token(token)
user_id = decoded_token['uid'] user_id = decoded_token["uid"]
user = User.query.filter_by(uid=user_id).first() user = User.query.filter_by(uid=user_id).first()
if user is None: if user is None:
return jsonify({'message': 'Error'}), 500 return jsonify({"message": "Error"}), 500
rating_count = Rating.query.filter_by(user_id=user_id).count() rating_count = Rating.query.filter_by(user_id=user_id).count()
return jsonify({'message': 'Ratings counted!', 'rating_count': rating_count}), 200 return jsonify(
{"message": "Ratings counted!", "rating_count": rating_count}
), 200
except Exception as e: except Exception as e:
print(e) print(e)
return jsonify({'message': str(e)}), 500 return jsonify({"message": str(e)}), 500
@app.route('/movie/<int:movie_id>', methods=['GET']) @app.route("/movie/<int:movie_id>", methods=["GET"])
def get_tmdb_data_movie_id(movie_id): def get_tmdb_data_movie_id(movie_id):
url = f'https://api.themoviedb.org/3/movie/{movie_id}' url = f"https://api.themoviedb.org/3/movie/{movie_id}"
headers = { headers = {"Authorization": f"Bearer {TMDB_BEARER_TOKEN}"}
'Authorization': f'Bearer {TMDB_BEARER_TOKEN}'}
try: try:
response = requests.get(url, headers=headers) response = requests.get(url, headers=headers)
@ -93,20 +97,19 @@ def get_tmdb_data_movie_id(movie_id):
return jsonify(response.json()), 200 return jsonify(response.json()), 200
except Exception as e: except Exception as e:
print(e) print(e)
return jsonify({'message': str(e)}), 500 return jsonify({"message": str(e)}), 500
@app.route('/movie', methods=['GET']) @app.route("/movie", methods=["GET"])
def get_tmdb_data_query(): def get_tmdb_data_query():
query = request.args.get('query') query = request.args.get("query")
if query: if query:
url = f'https://api.themoviedb.org/3/search/movie?query={query}' url = f"https://api.themoviedb.org/3/search/movie?query={query}"
else: else:
url = 'https://api.themoviedb.org/3/trending/movie/day' url = "https://api.themoviedb.org/3/trending/movie/day"
headers = { headers = {"Authorization": f"Bearer {TMDB_BEARER_TOKEN}"}
'Authorization': f'Bearer {TMDB_BEARER_TOKEN}'}
try: try:
response = requests.get(url, headers=headers) response = requests.get(url, headers=headers)
@ -114,88 +117,90 @@ def get_tmdb_data_query():
return jsonify(response.json()), 200 return jsonify(response.json()), 200
except Exception as e: except Exception as e:
print(e) print(e)
return jsonify({'message': str(e)}), 500 return jsonify({"message": str(e)}), 500
@app.route('/rating', methods=['POST']) @app.route("/rating", methods=["POST"])
def add_rating(): def add_rating():
token = request.json.get('token') token = request.json.get("token")
movie = request.json.get('movie') movie = request.json.get("movie")
value = request.json.get('value') value = request.json.get("value")
try: try:
decoded_token = auth.verify_id_token(token) decoded_token = auth.verify_id_token(token)
user_id = decoded_token['uid'] user_id = decoded_token["uid"]
# user = User.query.filter_by(uid=user_id).first() # user = User.query.filter_by(uid=user_id).first()
# if user is None: # if user is None:
# return jsonify({'message': 'Error'}), 500 # return jsonify({'message': 'Error'}), 500
rating = Rating.query.filter_by( rating = Rating.query.filter_by(user_id=user_id, movie_id=movie).first()
user_id=user_id, movie_id=movie).first()
if rating is None: if rating is None:
rating = Rating(user_id=user_id, movie_id=movie, value=value) rating = Rating(user_id=user_id, movie_id=movie, value=value)
db.session.add(rating) db.session.add(rating)
db.session.commit() db.session.commit()
return jsonify({'message': 'Rating added successfully!'}), 201 return jsonify({"message": "Rating added successfully!"}), 201
else: rating.value = value
rating.value = value
db.session.commit() db.session.commit()
return jsonify({'message': 'Rating updated successfully!'}), 200 return jsonify({"message": "Rating updated successfully!"}), 200
except Exception as e: except Exception as e:
print(e) print(e)
return jsonify({'message': str(e)}), 500 return jsonify({"message": str(e)}), 500
@app.route('/rating', methods=['DELETE']) @app.route("/rating", methods=["DELETE"])
def remove_rating(): def remove_rating():
token = request.json.get('token') token = request.json.get("token")
movie = request.json.get('movie') movie = request.json.get("movie")
try: try:
decoded_token = auth.verify_id_token(token) decoded_token = auth.verify_id_token(token)
user_id = decoded_token['uid'] user_id = decoded_token["uid"]
# user = User.query.filter_by(uid=user_id).first() # user = User.query.filter_by(uid=user_id).first()
# if user is None: # if user is None:
# return jsonify({'message': 'Error'}), 500 # return jsonify({'message': 'Error'}), 500
rating = Rating.query.filter_by( rating = Rating.query.filter_by(user_id=user_id, movie_id=movie).first()
user_id=user_id, movie_id=movie).first()
if rating is None: if rating is None:
return jsonify({'message': 'Error'}), 500 return jsonify({"message": "Error"}), 500
db.session.delete(rating) db.session.delete(rating)
db.session.commit() db.session.commit()
return jsonify({'message': 'Rating removed successfully!'}), 200 return jsonify({"message": "Rating removed successfully!"}), 200
except Exception as e: except Exception as e:
print(e) print(e)
return jsonify({'message': str(e)}), 500 return jsonify({"message": str(e)}), 500
@app.route('/get_rating', methods=['POST']) @app.route("/get_rating", methods=["POST"])
def get_rating(): def get_rating():
token = request.json.get('token') token = request.json.get("token")
movie = request.json.get('movie') movie = request.json.get("movie")
try: try:
decoded_token = auth.verify_id_token(token) decoded_token = auth.verify_id_token(token)
user_id = decoded_token['uid'] user_id = decoded_token["uid"]
# user = User.query.filter_by(uid=user_id).first() # user = User.query.filter_by(uid=user_id).first()
# if user is None: # if user is None:
# return jsonify({'message': 'Error'}), 500 # return jsonify({'message': 'Error'}), 500
rating = Rating.query.filter_by( rating = Rating.query.filter_by(user_id=user_id, movie_id=movie).first()
user_id=user_id, movie_id=movie).first()
if rating is None: if rating is None:
return jsonify({'message': 'Rating not found!'}), 200 return jsonify({"message": "Rating not found!"}), 200
else: return jsonify(
return jsonify({'message': 'Rating found!', 'movie': rating.movie_id, 'value': rating.value}), 200 {
"message": "Rating found!",
"movie": rating.movie_id,
"value": rating.value,
}
), 200
except Exception as e: except Exception as e:
print(e) print(e)
return jsonify({'message': str(e)}), 500 return jsonify({"message": str(e)}), 500
app.run(host="0.0.0.0", port=8084) app.run(host="0.0.0.0", port=8084)

72
Programming/ERSMS-project/connector/Include/frontend_AI_connector.py
View File

@ -1,14 +1,15 @@
from flask import Flask, request, jsonify
from flask_caching import Cache
import psycopg2
import pandas
import json import json
from configparser import ConfigParser from configparser import ConfigParser
from datetime import datetime from datetime import datetime
import pandas
import psycopg2
import requests import requests
from flask import Flask, jsonify
from flask_caching import Cache
app = Flask(__name__) app = Flask(__name__)
cache = Cache(config={'CACHE_TYPE': 'SimpleCache'}) cache = Cache(config={"CACHE_TYPE": "SimpleCache"})
db_connector = None db_connector = None
conn = None conn = None
movie_list = None movie_list = None
@ -19,7 +20,9 @@ def error_decorator(fun):
try: try:
fun(*args, **kwargs) fun(*args, **kwargs)
except psycopg2.DatabaseError: except psycopg2.DatabaseError:
return jsonify({"status": "Something... unexpected has occurred :sweat_smile:"}), 500 return jsonify(
{"status": "Something... unexpected has occurred :sweat_smile:"}
), 500
return inner1 return inner1
@ -34,7 +37,7 @@ def hello():
@app.route("/api/v3/get/<string:username>", methods=["GET"]) @app.route("/api/v3/get/<string:username>", methods=["GET"])
def access_user(username): def access_user(username):
cursor = conn.cursor() cursor = conn.cursor()
cursor.execute("select * from users where username='{}';".format(username)) cursor.execute(f"select * from users where username='{username}';")
res = cursor.fetchall() res = cursor.fetchall()
cursor.close() cursor.close()
@ -47,16 +50,16 @@ def access_user(username):
@app.route("/api/v3/add/<string:oauth_ID>/<string:username>", methods=["POST"]) @app.route("/api/v3/add/<string:oauth_ID>/<string:username>", methods=["POST"])
def add_user(oauth_ID, username): def add_user(oauth_ID, username):
cursor = conn.cursor() cursor = conn.cursor()
cursor.execute("select * from users where username='{}';".format(username)) cursor.execute(f"select * from users where username='{username}';")
res = cursor.fetchall() res = cursor.fetchall()
if len(res): if len(res):
cursor.close() cursor.close()
return jsonify({"status": "User already exists"}), 409 return jsonify({"status": "User already exists"}), 409
cursor.execute("INSERT INTO users (username, oauth_ID) VALUES ('{}','{}');".format( cursor.execute(
username, oauth_ID f"INSERT INTO users (username, oauth_ID) VALUES ('{username}','{oauth_ID}');"
)) )
conn.commit() conn.commit()
cursor.close() cursor.close()
@ -71,50 +74,54 @@ def get_recommendations(oauth_ID):
cursor.execute("select movie_ID from ratings where oauth_ID='{}'", oauth_ID) cursor.execute("select movie_ID from ratings where oauth_ID='{}'", oauth_ID)
res = cursor.fetchall() res = cursor.fetchall()
movies = [int(i) for i in res[0]] movies = [int(i) for i in res[0]]
url = 'http://localhost:8081/api/v3/AI_recommendations' url = "http://localhost:8081/api/v3/AI_recommendations"
response = requests.post(url, response = requests.post(
json=movies, url, json=movies, headers={"Content-Type": "application/json"}
headers={'Content-Type': 'application/json'}) )
return jsonify(response.json()), 200 return jsonify(response.json()), 200
@app.route("/api/v3/get_movie/<int:movie_ID>", methods=["GET"]) @app.route("/api/v3/get_movie/<int:movie_ID>", methods=["GET"])
def get_movie(movie_ID): def get_movie(movie_ID):
movie_info = movie_list.loc[movie_list['movie_id'] == movie_ID] movie_info = movie_list.loc[movie_list["movie_id"] == movie_ID]
if movie_info.empty: if movie_info.empty:
return jsonify({"status": "Movie with ID {} doesn't exist".format(movie_ID)} return jsonify({"status": f"Movie with ID {movie_ID} doesn't exist"}), 404
), 404
cast = json.loads(movie_info["cast"][0].replace('\\"', '"')) cast = json.loads(movie_info["cast"][0].replace('\\"', '"'))
crew = json.loads(movie_info["crew"][0].replace('\\"', '"')) crew = json.loads(movie_info["crew"][0].replace('\\"', '"'))
output_json = {"movie_id": movie_ID, output_json = {
"title": movie_info["title"][0], "movie_id": movie_ID,
"cast": cast, "title": movie_info["title"][0],
"crew": crew} "cast": cast,
"crew": crew,
}
return jsonify(output_json), 200 return jsonify(output_json), 200
@app.route("/api/v3/rate_movie/<string:uID>/<string:movie_ID>/<int:rating>", methods=["POST"]) @app.route(
"/api/v3/rate_movie/<string:uID>/<string:movie_ID>/<int:rating>", methods=["POST"]
)
def rate_movie(uID, movie_ID, rating): def rate_movie(uID, movie_ID, rating):
movie_info = movie_list.loc[movie_list['movie_id'] == int(movie_ID)] movie_info = movie_list.loc[movie_list["movie_id"] == int(movie_ID)]
if movie_info.empty: if movie_info.empty:
return jsonify({"status": "Movie with ID {} doesn't exist".format(movie_ID)} return jsonify({"status": f"Movie with ID {movie_ID} doesn't exist"}), 404
), 404
if rating < 1 or rating > 5: if rating < 1 or rating > 5:
return jsonify({"status": "Incorrect rating"}), 400 return jsonify({"status": "Incorrect rating"}), 400
cursor = conn.cursor() cursor = conn.cursor()
cursor.execute("select * from users where oauth_ID='{}';".format(uID)) cursor.execute(f"select * from users where oauth_ID='{uID}';")
res = cursor.fetchall() res = cursor.fetchall()
if not len(res): if not len(res):
cursor.close() cursor.close()
return jsonify({"status": "User doesn't exists"}), 404 return jsonify({"status": "User doesn't exists"}), 404
cursor.execute("select * from ratings where oauth_ID='{}' AND movie_ID='{}';".format(uID, movie_ID)) cursor.execute(
f"select * from ratings where oauth_ID='{uID}' AND movie_ID='{movie_ID}';"
)
res = cursor.fetchall() res = cursor.fetchall()
if len(res): if len(res):
@ -126,9 +133,9 @@ def rate_movie(uID, movie_ID, rating):
""" """
cursor.execute(sql.format(rating, uID, movie_ID)) cursor.execute(sql.format(rating, uID, movie_ID))
else: else:
cursor.execute("INSERT INTO ratings (movie_ID, oauth_ID, rating) VALUES ('{}','{}',{});".format( cursor.execute(
movie_ID, uID, rating f"INSERT INTO ratings (movie_ID, oauth_ID, rating) VALUES ('{movie_ID}','{uID}',{rating});"
)) )
conn.commit() conn.commit()
cursor.close() cursor.close()
@ -147,7 +154,7 @@ if __name__ == "__main__":
database=config["postgres"]["database"], database=config["postgres"]["database"],
user=config["postgres"]["user"], user=config["postgres"]["user"],
password=config["postgres"]["password"], password=config["postgres"]["password"],
port=int(config["postgres"]["port"]) port=int(config["postgres"]["port"]),
) )
except Exception: except Exception:
@ -161,4 +168,3 @@ if __name__ == "__main__":
app.run(host="localhost", port=8090, debug=True) app.run(host="localhost", port=8090, debug=True)
conn.close() conn.close()

96
Programming/ERSMS-project/movie_recommendations/movie_recommender.py
View File

@ -1,32 +1,32 @@
import pandas as pd
import numpy as np
from ast import literal_eval
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import hashlib import hashlib
import json import json
from ast import literal_eval
from configparser import ConfigParser from configparser import ConfigParser
import psycopg2
from flask import Flask, request, jsonify
from flask_caching import Cache
import numpy as np
import pandas as pd
import psycopg2
from flask import Flask, jsonify, request
from flask_caching import Cache
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import cosine_similarity
app = Flask(__name__) app = Flask(__name__)
cache = Cache(config={'CACHE_TYPE': 'SimpleCache'}) cache = Cache(config={"CACHE_TYPE": "SimpleCache"})
db_connector = None db_connector = None
conn = None conn = None
def get_director(x): def get_director(x):
for i in x: for i in x:
if i['job'] == 'Director': if i["job"] == "Director":
return i['name'] return i["name"]
return np.nan return np.nan
def get_list(x): def get_list(x):
if isinstance(x, list): if isinstance(x, list):
names = [i['name'] for i in x] names = [i["name"] for i in x]
if len(names) > 3: if len(names) > 3:
names = names[:3] names = names[:3]
return names return names
@ -36,15 +36,21 @@ def get_list(x):
def clean_data(x): def clean_data(x):
if isinstance(x, list): if isinstance(x, list):
return [str.lower(i.replace(" ", "")) for i in x] return [str.lower(i.replace(" ", "")) for i in x]
else: if isinstance(x, str):
if isinstance(x, str): return str.lower(x.replace(" ", ""))
return str.lower(x.replace(" ", "")) return ""
else:
return ''
def create_soup(x): def create_soup(x):
return ' '.join(x['keywords']) + ' ' + ' '.join(x['cast']) + ' ' + x['director'] + ' ' + ' '.join(x['genres']) return (
" ".join(x["keywords"])
+ " "
+ " ".join(x["cast"])
+ " "
+ x["director"]
+ " "
+ " ".join(x["genres"])
)
class MovieRecommender: class MovieRecommender:
@ -53,61 +59,58 @@ class MovieRecommender:
self.cosine_sim = None self.cosine_sim = None
def fit(self, credits_file, movies_file): def fit(self, credits_file, movies_file):
""" """Fittuje AI do przekazanych danych
Fittuje AI do przekazanych danych
:param credits_file: csv z creditsami :param credits_file: csv z creditsami
:param movies_file: csv z filmami :param movies_file: csv z filmami
:return: Nic :return: Nic.
""" """
df1 = pd.read_csv(credits_file) df1 = pd.read_csv(credits_file)
df2 = pd.read_csv(movies_file) df2 = pd.read_csv(movies_file)
df1.columns = ['id', 'tittle', 'cast', 'crew'] df1.columns = ["id", "tittle", "cast", "crew"]
df2 = df2.merge(df1, on='id') df2 = df2.merge(df1, on="id")
df2['overview'] = df2['overview'].fillna('') df2["overview"] = df2["overview"].fillna("")
self.df = df2 self.df = df2
features = ['cast', 'crew', 'keywords', 'genres'] features = ["cast", "crew", "keywords", "genres"]
for feature in features: for feature in features:
df2[feature] = df2[feature].apply(literal_eval) df2[feature] = df2[feature].apply(literal_eval)
df2['director'] = df2['crew'].apply(get_director) df2["director"] = df2["crew"].apply(get_director)
features = ['cast', 'keywords', 'genres'] features = ["cast", "keywords", "genres"]
for feature in features: for feature in features:
df2[feature] = df2[feature].apply(get_list) df2[feature] = df2[feature].apply(get_list)
features = ['cast', 'keywords', 'director', 'genres'] features = ["cast", "keywords", "director", "genres"]
for feature in features: for feature in features:
df2[feature] = df2[feature].apply(clean_data) df2[feature] = df2[feature].apply(clean_data)
df2['soup'] = df2.apply(create_soup, axis=1) df2["soup"] = df2.apply(create_soup, axis=1)
count = CountVectorizer(stop_words='english') count = CountVectorizer(stop_words="english")
count_matrix = count.fit_transform(df2['soup']) count_matrix = count.fit_transform(df2["soup"])
self.cosine_sim = cosine_similarity(count_matrix, count_matrix) self.cosine_sim = cosine_similarity(count_matrix, count_matrix)
self.df = df2.reset_index() self.df = df2.reset_index()
def _get_recommendations_one_input(self, movie_id): def _get_recommendations_one_input(self, movie_id):
""" """Tworzy rekomendacje, bazując na jednym filmie
Tworzy rekomendacje, bazując na jednym filmie
:param movie_id: id filmu, dla którego ma zrobić rekomendację :param movie_id: id filmu, dla którego ma zrobić rekomendację
:return: Zwraca listę [movie_ids, similarity_scores] gdzie oba argumenty są np.array :return: Zwraca listę [movie_ids, similarity_scores] gdzie oba argumenty są np.array.
""" """
indices = pd.Series(self.df.index, index=self.df['id']).drop_duplicates() indices = pd.Series(self.df.index, index=self.df["id"]).drop_duplicates()
idx = indices[movie_id] idx = indices[movie_id]
sim_scores = list(enumerate(self.cosine_sim[idx])) sim_scores = list(enumerate(self.cosine_sim[idx]))
sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True) sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)
sim_scores = sim_scores[1:101] sim_scores = sim_scores[1:101]
movie_indices = [i[0] for i in sim_scores] movie_indices = [i[0] for i in sim_scores]
sim_scores = np.array([t[1] for t in sim_scores]) sim_scores = np.array([t[1] for t in sim_scores])
return [self.df['id'].iloc[movie_indices].values, sim_scores] return [self.df["id"].iloc[movie_indices].values, sim_scores]
def get_recommendations(self, movie_ids: list) -> {}: def get_recommendations(self, movie_ids: list) -> {}:
""" """Tworzy listę rekomendacji bazującą na id podanych filmów
Tworzy listę rekomendacji bazującą na id podanych filmów
:param movie_ids: id filmów, na podstawie których ma wybrać rekomendowane filmy :param movie_ids: id filmów, na podstawie których ma wybrać rekomendowane filmy
:return: Zwraca dicta {movie_id: similarity_scores} :return: Zwraca dicta {movie_id: similarity_scores}.
""" """
recommended_movies = {} recommended_movies = {}
for movie_id in movie_ids: for movie_id in movie_ids:
@ -117,22 +120,25 @@ class MovieRecommender:
continue continue
if recommended_movies.get(int(recommended_id)) is None: if recommended_movies.get(int(recommended_id)) is None:
recommended_movies[int(recommended_id)] = float(round((sim_score / len(movie_ids)), 4)) recommended_movies[int(recommended_id)] = float(
round((sim_score / len(movie_ids)), 4)
)
else: else:
recommended_movies[int(recommended_id)] += float(round((sim_score / len(movie_ids)), 4)) recommended_movies[int(recommended_id)] += float(
round((sim_score / len(movie_ids)), 4)
)
return recommended_movies return recommended_movies
recommender = MovieRecommender() recommender = MovieRecommender()
recommender.fit('datasets/tmdb_5000_credits.csv', recommender.fit("datasets/tmdb_5000_credits.csv", "datasets/tmdb_5000_movies.csv")
'datasets/tmdb_5000_movies.csv')
def make_cache_key(): def make_cache_key():
data = request.get_json() data = request.get_json()
if isinstance(data, list): if isinstance(data, list):
data = sorted(data) data = sorted(data)
key = hashlib.md5(json.dumps(data).encode('utf-8')).hexdigest() key = hashlib.md5(json.dumps(data).encode("utf-8")).hexdigest()
return key return key
@ -155,7 +161,7 @@ if __name__ == "__main__":
database=config["postgres"]["database"], database=config["postgres"]["database"],
user=config["postgres"]["user"], user=config["postgres"]["user"],
password=config["postgres"]["password"], password=config["postgres"]["password"],
port=int(config["postgres"]["port"]) port=int(config["postgres"]["port"]),
) )
except Exception: except Exception:

72
Programming/NLP/compare_chunks.py
View File

@ -1,58 +1,65 @@
def reformat_sentence(sentence): def reformat_sentence(sentence):
""" """Reformats a sentence by replacing slashes '/' with ' ]'."""
Reformats a sentence by replacing slashes '/' with ' ]'. return sentence.replace("/", " ]")
"""
return sentence.replace('/', ' ]')
def insert_commas(sentence, reference_sentence): def insert_commas(sentence, reference_sentence):
""" """Inserts commas into 'sentence' at the same positions as they appear in 'reference_sentence'."""
Inserts commas into 'sentence' at the same positions as they appear in 'reference_sentence'.
"""
# Splitting the sentences into chunks # Splitting the sentences into chunks
sentence_chunks = sentence.split(' ] [ ') sentence_chunks = sentence.split(" ] [ ")
reference_chunks = reference_sentence.split(' ] [ ') reference_chunks = reference_sentence.split(" ] [ ")
# Insert commas into the original sentence based on the reference sentence # Insert commas into the original sentence based on the reference sentence
for i, chunk in enumerate(reference_chunks): for i, chunk in enumerate(reference_chunks):
if ',' in chunk and i < len(sentence_chunks): if "," in chunk and i < len(sentence_chunks):
sentence_chunks[i] += ',' sentence_chunks[i] += ","
# Reconstruct the sentence with inserted commas # Reconstruct the sentence with inserted commas
return ' ] [ '.join(sentence_chunks) return " ] [ ".join(sentence_chunks)
def process_sentences(sentences1, sentences2): def process_sentences(sentences1, sentences2):
""" """Processes two lists of sentences according to the specified rules."""
Processes two lists of sentences according to the specified rules.
"""
processed_sentences = [] processed_sentences = []
for sentence1, sentence2 in zip(sentences1, sentences2): for sentence1, sentence2 in zip(sentences1, sentences2):
# Reformat the first sentence # Reformat the first sentence
reformatted_sentence1 = reformat_sentence(sentence1) reformatted_sentence1 = reformat_sentence(sentence1)
reformatted_sentence1 = insert_commas(reformatted_sentence1, sentence2) reformatted_sentence1 = insert_commas(reformatted_sentence1, sentence2)
# Splitting the sentences into words, ignoring the square brackets # Splitting the sentences into words, ignoring the square brackets
words1 = set(' '.join(reformatted_sentence1.strip().split('] [')).replace('[', '').replace(']', '').lower().split()) words1 = set(
words2 = set(' '.join(sentence2.strip().split('] [')).replace('[', '').replace(']', '').lower().split()) " ".join(reformatted_sentence1.strip().split("] ["))
.replace("[", "")
.replace("]", "")
.lower()
.split()
)
words2 = set(
" ".join(sentence2.strip().split("] ["))
.replace("[", "")
.replace("]", "")
.lower()
.split()
)
# Finding differences in words # Finding differences in words
diff = words2 - words1 diff = words2 - words1
# Preparing the output sentence # Preparing the output sentence
output_sentence = reformatted_sentence1.strip() output_sentence = reformatted_sentence1.strip()
if '[ . ]' in sentence2: if "[ . ]" in sentence2:
output_sentence += ' [ . ]' output_sentence += " [ . ]"
# Adding the differences and the output sentence to the processed_sentences list # Adding the differences and the output sentence to the processed_sentences list
processed_sentences.append({ processed_sentences.append(
"sentence1": reformatted_sentence1.strip(), {
"sentence2": sentence2.strip(), "sentence1": reformatted_sentence1.strip(),
"differences": diff, "sentence2": sentence2.strip(),
"merged_sentence": output_sentence "differences": diff,
}) "merged_sentence": output_sentence,
if diff and diff != {'.'}: }
)
if diff and diff != {"."}:
print("Difference found!") print("Difference found!")
print("Sentence 1:", reformatted_sentence1.strip()) print("Sentence 1:", reformatted_sentence1.strip())
print("Sentence 2:", sentence2.strip()) print("Sentence 2:", sentence2.strip())
@ -61,12 +68,13 @@ def process_sentences(sentences1, sentences2):
return processed_sentences return processed_sentences
# First, let's open and read the contents of both files to understand their structure and content. # First, let's open and read the contents of both files to understand their structure and content.
file_path_1 = 'test_goldStandard/images/images-chunks-gpt-one.txt' file_path_1 = "test_goldStandard/images/images-chunks-gpt-one.txt"
file_path_2 = 'test_goldStandard/images/STSint.testinput.images.sent1.chunk.txt' file_path_2 = "test_goldStandard/images/STSint.testinput.images.sent1.chunk.txt"
# Reading the files # Reading the files
with open(file_path_1, 'r') as file1, open(file_path_2, 'r') as file2: with open(file_path_1) as file1, open(file_path_2) as file2:
sentences_file_1 = file1.readlines() sentences_file_1 = file1.readlines()
sentences_file_2 = file2.readlines() sentences_file_2 = file2.readlines()

25
Programming/NLP/create_alignments.py
View File

@ -1,14 +1,14 @@
import processing
import pandas as pd
import gpt_alignment import gpt_alignment
import pandas as pd
import processing
# Specify the output file path # Specify the output file path
file_path = "alignments_unformatted_headlines.txt" file_path = "alignments_unformatted_headlines.txt"
# paths to students answers database # paths to students answers database
#chunked_path1 = "test_goldStandard/student/STSint.testinput.answers-students.sent1.chunk.txt" # chunked_path1 = "test_goldStandard/student/STSint.testinput.answers-students.sent1.chunk.txt"
#chunked_path2 = "test_goldStandard/student/STSint.testinput.answers-students.sent2.chunk.txt" # chunked_path2 = "test_goldStandard/student/STSint.testinput.answers-students.sent2.chunk.txt"
#alignment_path = "test_goldStandard/student/STSint.testinput.answers-students.wa" # alignment_path = "test_goldStandard/student/STSint.testinput.answers-students.wa"
# paths to headlines # paths to headlines
chunked_path1 = "test_goldStandard/headlines/STSint.testinput.headlines.sent1.chunk.txt" chunked_path1 = "test_goldStandard/headlines/STSint.testinput.headlines.sent1.chunk.txt"
@ -16,22 +16,23 @@ chunked_path2 = "test_goldStandard/headlines/STSint.testinput.headlines.sent1.ch
alignment_path = "test_goldStandard/headlines/STSint.testinput.headlines.wa" alignment_path = "test_goldStandard/headlines/STSint.testinput.headlines.wa"
# load data # load data
#studentAnserws = processing.load_sentences(studentAnswers1_path, studentAnswers1_path) # studentAnserws = processing.load_sentences(studentAnswers1_path, studentAnswers1_path)
goldstandard_chunked = processing.load_chunked(chunked_path1, chunked_path2) goldstandard_chunked = processing.load_chunked(chunked_path1, chunked_path2)
goldstandard_alignment = processing.load_alignment(alignment_path) goldstandard_alignment = processing.load_alignment(alignment_path)
# get a nice anwser-student table # get a nice anwser-student table
data = pd.merge(goldstandard_chunked, goldstandard_alignment, left_index=True, right_index=True) data = pd.merge(
#print(data) goldstandard_chunked, goldstandard_alignment, left_index=True, right_index=True
)
# print(data)
data_for_chat, indexes = processing.get_chunks_as_text(data) data_for_chat, indexes = processing.get_chunks_as_text(data)
client = gpt_alignment.createGPT() client = gpt_alignment.createGPT()
responses = [] responses = []
for i in range(0, len(data_for_chat)): for i in range(len(data_for_chat)):
responses.append(gpt_alignment.callApi(client, data_for_chat[i])) responses.append(gpt_alignment.callApi(client, data_for_chat[i]))
# Writing to the file with repr() to preserve "\n" characters # Writing to the file with repr() to preserve "\n" characters
with open(file_path, 'w') as file: with open(file_path, "w") as file:
for string in responses: file.writelines(repr(string) + "\n" for string in responses)
file.write(repr(string) + '\n')

30
Programming/NLP/create_alignments_with_examples.py
View File

@ -1,14 +1,14 @@
import processing
import pandas as pd
import gpt_alignment import gpt_alignment
import pandas as pd
import processing
# Specify the output file path # Specify the output file path
file_path = "alignments_with_training_headlines2.wa" file_path = "alignments_with_training_headlines2.wa"
# paths to students andsewrs database # paths to students andsewrs database
#chunked_path1 = "test_goldStandard/student/STSint.testinput.answers-students.sent1.chunk.txt" # chunked_path1 = "test_goldStandard/student/STSint.testinput.answers-students.sent1.chunk.txt"
#chunked_path2 = "test_goldStandard/student/STSint.testinput.answers-students.sent2.chunk.txt" # chunked_path2 = "test_goldStandard/student/STSint.testinput.answers-students.sent2.chunk.txt"
#alignment_path = "test_goldStandard/student/STSint.testinput.answers-students.wa" # alignment_path = "test_goldStandard/student/STSint.testinput.answers-students.wa"
# paths to headlines # paths to headlines
chunked_path1 = "test_goldStandard/headlines/STSint.testinput.headlines.sent1.chunk.txt" chunked_path1 = "test_goldStandard/headlines/STSint.testinput.headlines.sent1.chunk.txt"
@ -20,26 +20,30 @@ goldstandard_chunked = processing.load_chunked(chunked_path1, chunked_path2)
goldstandard_alignment = processing.load_alignment(alignment_path) goldstandard_alignment = processing.load_alignment(alignment_path)
# get a nice anwser-student table # get a nice anwser-student table
data = pd.merge(goldstandard_chunked, goldstandard_alignment, left_index=True, right_index=True) data = pd.merge(
#print(data) goldstandard_chunked, goldstandard_alignment, left_index=True, right_index=True
)
# print(data)
train, test = processing.generate_train_test_split(data) train, test = processing.generate_train_test_split(data)
data_for_chat, indexes = processing.get_chunks_as_text(test) data_for_chat, indexes = processing.get_chunks_as_text(test)
_, indexes_of_training = processing.get_chunks_as_text(train) _, indexes_of_training = processing.get_chunks_as_text(train)
indexes_of_training = [i+1 for i in indexes_of_training] indexes_of_training = [i + 1 for i in indexes_of_training]
indexes = [i+1 for i in indexes] indexes = [i + 1 for i in indexes]
print(indexes_of_training) print(indexes_of_training)
print(indexes) print(indexes)
client = gpt_alignment.createGPT() client = gpt_alignment.createGPT()
responses = [] responses = []
for i in range(0, 10):#len(data_for_chat)): for i in range(10): # len(data_for_chat)):
responses.append([gpt_alignment.callApi_examples(client, train, data_for_chat[i]), indexes[i]]) responses.append(
[gpt_alignment.callApi_examples(client, train, data_for_chat[i]), indexes[i]]
)
with open(file_path, 'w') as file: with open(file_path, "w") as file:
for i, r in enumerate(responses): for i, r in enumerate(responses):
file.write("<sentence id=\"" + str(r[1]+1) + "\" status=\"\">\n") file.write('<sentence id="' + str(r[1] + 1) + '" status="">\n')
file.write("<alignment>\n") file.write("<alignment>\n")
file.write(r[0]) file.write(r[0])
file.write("\n</alignment>\n") file.write("\n</alignment>\n")

73
Programming/NLP/format_alignments.py
View File

@ -1,7 +1,8 @@
import copy
import re
import pandas as pd import pandas as pd
import processing import processing
import re
import copy
# input file name # input file name
file_path = "alignments_unformatted_headlines.txt" file_path = "alignments_unformatted_headlines.txt"
@ -15,20 +16,22 @@ chunked_path2 = "test_goldStandard/headlines/STSint.testinput.headlines.sent1.ch
alignment_path = "test_goldStandard/headlines/STSint.testinput.headlines.wa" alignment_path = "test_goldStandard/headlines/STSint.testinput.headlines.wa"
# paths to students andsewrs database # paths to students andsewrs database
#chunked_path1 = "test_goldStandard/student/STSint.testinput.answers-students.sent1.chunk.txt" # chunked_path1 = "test_goldStandard/student/STSint.testinput.answers-students.sent1.chunk.txt"
#chunked_path2 = "test_goldStandard/student/STSint.testinput.answers-students.sent2.chunk.txt" # chunked_path2 = "test_goldStandard/student/STSint.testinput.answers-students.sent2.chunk.txt"
#alignment_path = "test_goldStandard/student/STSint.testinput.answers-students.wa" # alignment_path = "test_goldStandard/student/STSint.testinput.answers-students.wa"
# load data # load data
goldstandard_chunked = processing.load_chunked(chunked_path1, chunked_path2) goldstandard_chunked = processing.load_chunked(chunked_path1, chunked_path2)
goldstandard_alignment = processing.load_alignment(alignment_path) goldstandard_alignment = processing.load_alignment(alignment_path)
# get a nice table # get a nice table
data = pd.merge(goldstandard_chunked, goldstandard_alignment, left_index=True, right_index=True) data = pd.merge(
goldstandard_chunked, goldstandard_alignment, left_index=True, right_index=True
)
# open generated alignments # open generated alignments
with open(file_path, 'r') as file: with open(file_path) as file:
responses = [eval(line.strip()) for line in file.readlines()] responses = [eval(line.strip()) for line in file]
for i, r in enumerate(responses): for i, r in enumerate(responses):
print("\nresponse number " + str(i)) print("\nresponse number " + str(i))
@ -40,12 +43,12 @@ for i, response in enumerate(responses):
temp = response.lstrip("\n") temp = response.lstrip("\n")
temp = temp.rstrip("\n") temp = temp.rstrip("\n")
temp = re.sub(r'(?<==)>', '> ', temp) # add space after > temp = re.sub(r"(?<==)>", "> ", temp) # add space after >
temp = re.sub(r'(?<!\n) +(?!\n)', ' ', temp) # remove double space temp = re.sub(r"(?<!\n) +(?!\n)", " ", temp) # remove double space
temp = temp.replace("]", "") temp = temp.replace("]", "")
temp = temp.replace("[", "") temp = temp.replace("[", "")
temp = temp.replace("\'", "") temp = temp.replace("'", "")
temp = temp.replace(" ==> ", " <==> ") temp = temp.replace(" ==> ", " <==> ")
temp = temp.replace(" => ", " <==> ") temp = temp.replace(" => ", " <==> ")
temp = temp.replace(" <=> ", " <==> ") temp = temp.replace(" <=> ", " <==> ")
@ -56,7 +59,7 @@ for i, response in enumerate(responses):
temp = temp.replace("\n// NOALI", "\n0 <==> 0 // NOALI") temp = temp.replace("\n// NOALI", "\n0 <==> 0 // NOALI")
temp = temp.replace("\n // ", "\n0 <==> ") temp = temp.replace("\n // ", "\n0 <==> ")
temp = temp.replace("\n// ", "\n0 <==> ") temp = temp.replace("\n// ", "\n0 <==> ")
temp = re.sub(r'(^|[^<])(==>+)', r' <==>', temp) temp = re.sub(r"(^|[^<])(==>+)", r" <==>", temp)
temp = temp.replace("// - //", "// 0 //") temp = temp.replace("// - //", "// 0 //")
temp = temp.replace("// score", "// ") temp = temp.replace("// score", "// ")
temp = temp.replace("// alignment type", "// NOALI") temp = temp.replace("// alignment type", "// NOALI")
@ -64,15 +67,15 @@ for i, response in enumerate(responses):
temp = temp.replace("- <", "0 <") temp = temp.replace("- <", "0 <")
temp = temp.replace("// //", "// NOALI //") temp = temp.replace("// //", "// NOALI //")
temp = temp.replace("equi", "EQUI") temp = temp.replace("equi", "EQUI")
temp = re.sub(r'\d\. ', '', temp) # remove 1., 2. ... temp = re.sub(r"\d\. ", "", temp) # remove 1., 2. ...
temp = temp.upper() temp = temp.upper()
temp = re.sub(r'^(<==>)', r'0 \1', temp) temp = re.sub(r"^(<==>)", r"0 \1", temp)
temp = re.sub(r'(?<!\n) +(?!\n)', ' ', temp) # remove double space temp = re.sub(r"(?<!\n) +(?!\n)", " ", temp) # remove double space
temp = temp.split("\n") temp = temp.split("\n")
for k, t in enumerate(temp): for k, t in enumerate(temp):
if "<==>" not in temp[k]: if "<==>" not in t:
temp[k] = "0 <==> " + temp[k] temp[k] = "0 <==> " + temp[k]
temp[k] = temp[k].split("<==>") temp[k] = temp[k].split("<==>")
temp[k] = [temp[k][0], *temp[k][1].split("//")] temp[k] = [temp[k][0], *temp[k][1].split("//")]
@ -81,11 +84,11 @@ for i, response in enumerate(responses):
q = 1 q = 1
numberList = [] numberList = []
for chunk in chunk1arr: for chunk in chunk1arr:
chunk = re.sub(r'(?<!\n) +(?!\n)', ' ', chunk) chunk = re.sub(r"(?<!\n) +(?!\n)", " ", chunk)
n_of_words = len(chunk.strip().split(" ")) n_of_words = len(chunk.strip().split(" "))
index_str = "" index_str = ""
for qq in range(q, q + n_of_words): for qq in range(q, q + n_of_words):
index_str = index_str + str(qq) + " " index_str = index_str + str(qq) + " "
numberList.append(index_str) numberList.append(index_str)
q = q + n_of_words q = q + n_of_words
@ -97,11 +100,11 @@ for i, response in enumerate(responses):
q = 1 q = 1
numberList = [] numberList = []
for chunk in chunk1arr: for chunk in chunk1arr:
chunk = re.sub(r'(?<!\n) +(?!\n)', ' ', chunk) chunk = re.sub(r"(?<!\n) +(?!\n)", " ", chunk)
n_of_words = len(chunk.strip().split(" ")) n_of_words = len(chunk.strip().split(" "))
index_str = "" index_str = ""
for qq in range(q, q + n_of_words): for qq in range(q, q + n_of_words):
index_str = index_str + str(qq) + " " index_str = index_str + str(qq) + " "
numberList.append(index_str) numberList.append(index_str)
q = q + n_of_words q = q + n_of_words
@ -109,7 +112,9 @@ for i, response in enumerate(responses):
pattern = re.compile(chunk, re.IGNORECASE) pattern = re.compile(chunk, re.IGNORECASE)
temp[k][1] = pattern.sub(numberList[j], temp[k][1]) temp[k][1] = pattern.sub(numberList[j], temp[k][1])
if any(char.isalpha() and ord(char) < 128 for char in temp[k][0]) or any(char.isalpha() and ord(char) < 128 for char in temp[k][1]): if any(char.isalpha() and ord(char) < 128 for char in temp[k][0]) or any(
char.isalpha() and ord(char) < 128 for char in temp[k][1]
):
temp[k] = "" temp[k] = ""
if len(temp[k]) >= 4: if len(temp[k]) >= 4:
temp[k][3] = temp[k][3].replace("NOALI", "0") temp[k][3] = temp[k][3].replace("NOALI", "0")
@ -117,11 +122,19 @@ for i, response in enumerate(responses):
temp[k][3] = " 0 " temp[k][3] = " 0 "
if temp[k][3] == " ": if temp[k][3] == " ":
temp[k][3] = " 0 " temp[k][3] = " 0 "
temp[k] = temp[k][0] + " <==> " + temp[k][1] + " // " + temp[k][2] + " // " + temp[k][3] temp[k] = (
temp[k][0]
+ " <==> "
+ temp[k][1]
+ " // "
+ temp[k][2]
+ " // "
+ temp[k][3]
)
elif len(temp[k]) == 3: elif len(temp[k]) == 3:
temp[k] = temp[k][0] + " <==> " + temp[k][1] + " // " + temp[k][2] + " // 0" temp[k] = temp[k][0] + " <==> " + temp[k][1] + " // " + temp[k][2] + " // 0"
temp[k] = re.sub(r'\s{2,}', ' ', temp[k]) # remove double space temp[k] = re.sub(r"\s{2,}", " ", temp[k]) # remove double space
temp = [x for x in temp if x != ""] temp = [x for x in temp if x != ""]
responses[i] = "\n".join(temp).strip() responses[i] = "\n".join(temp).strip()
@ -130,21 +143,21 @@ indexes = []
responses_final = [] responses_final = []
rejected_indexes = [] rejected_indexes = []
for n, r in enumerate(responses): for n, r in enumerate(responses):
if r == '': if r == "":
rejected_indexes.append(n+1) rejected_indexes.append(n + 1)
continue continue
if r == '\n': if r == "\n":
rejected_indexes.append(n+1) rejected_indexes.append(n + 1)
continue continue
indexes.append(n+1) indexes.append(n + 1)
responses_final.append(r) responses_final.append(r)
print("rejected indexes:") print("rejected indexes:")
print(rejected_indexes) print(rejected_indexes)
with open(output_file_path, 'w') as file: with open(output_file_path, "w") as file:
for i, r in zip(indexes, responses_final): for i, r in zip(indexes, responses_final):
file.write("<sentence id=\"" + str(i+1) + "\" status=\"\">\n") file.write('<sentence id="' + str(i + 1) + '" status="">\n')
file.write("<alignment>\n") file.write("<alignment>\n")
file.write(r) file.write(r)
file.write("\n</alignment>\n") file.write("\n</alignment>\n")

87
Programming/NLP/format_alignments_with_training.py
View File

@ -1,22 +1,37 @@
import copy
import re
import pandas as pd import pandas as pd
import processing import processing
import re
import copy
# paths to students andsewrs database # paths to students andsewrs database
studentAnswers1_path = "test_goldStandard/student/STSint.testinput.answers-students.sent1.txt" studentAnswers1_path = (
studentAnswers2_path = "test_goldStandard/student/STSint.testinput.answers-students.sent2.txt" "test_goldStandard/student/STSint.testinput.answers-students.sent1.txt"
studentAnsewrs_chunked_path1 = "test_goldStandard/student/STSint.testinput.answers-students.sent1.chunk.txt" )
studentAnsewrs_chunked_path2 = "test_goldStandard/student/STSint.testinput.answers-students.sent2.chunk.txt" studentAnswers2_path = (
studentsAnsewrs_alignment_path = "test_goldStandard/student/STSint.testinput.answers-students.wa" "test_goldStandard/student/STSint.testinput.answers-students.sent2.txt"
)
studentAnsewrs_chunked_path1 = (
"test_goldStandard/student/STSint.testinput.answers-students.sent1.chunk.txt"
)
studentAnsewrs_chunked_path2 = (
"test_goldStandard/student/STSint.testinput.answers-students.sent2.chunk.txt"
)
studentsAnsewrs_alignment_path = (
"test_goldStandard/student/STSint.testinput.answers-students.wa"
)
# load data # load data
studentAnserws = processing.load_sentences(studentAnswers1_path, studentAnswers1_path) studentAnserws = processing.load_sentences(studentAnswers1_path, studentAnswers1_path)
goldstandard_chunked = processing.load_chunked(studentAnsewrs_chunked_path1, studentAnsewrs_chunked_path2) goldstandard_chunked = processing.load_chunked(
studentAnsewrs_chunked_path1, studentAnsewrs_chunked_path2
)
goldstandard_alignment = processing.load_alignment(studentsAnsewrs_alignment_path) goldstandard_alignment = processing.load_alignment(studentsAnsewrs_alignment_path)
# get a nice anwser-student table # get a nice anwser-student table
data = pd.merge(goldstandard_chunked, goldstandard_alignment, left_index=True, right_index=True) data = pd.merge(
goldstandard_chunked, goldstandard_alignment, left_index=True, right_index=True
)
file_path = "alignments_with_training_student_K.wa" file_path = "alignments_with_training_student_K.wa"
@ -35,8 +50,8 @@ for i, response in enumerate(responses):
temp = response.lstrip("\n") temp = response.lstrip("\n")
temp = temp.rstrip("\n") temp = temp.rstrip("\n")
temp = re.sub(r'(?<==)>', '> ', temp) # add space after > temp = re.sub(r"(?<==)>", "> ", temp) # add space after >
temp = re.sub(r'(?<!\n) +(?!\n)', ' ', temp) # remove double space temp = re.sub(r"(?<!\n) +(?!\n)", " ", temp) # remove double space
temp = temp.replace("]", "") temp = temp.replace("]", "")
temp = temp.replace("[", "") temp = temp.replace("[", "")
@ -49,32 +64,49 @@ for i, response in enumerate(responses):
temp = temp.replace("NOALI <==>", "0 <==>") temp = temp.replace("NOALI <==>", "0 <==>")
temp = temp.replace("<==> NOALI", "<==> 0") temp = temp.replace("<==> NOALI", "<==> 0")
temp = temp.replace("\n// NOALI", "\n0 <==> 0 // NOALI") temp = temp.replace("\n// NOALI", "\n0 <==> 0 // NOALI")
temp = re.sub(r'(^|[^<])(==>+)', r' <==>', temp) temp = re.sub(r"(^|[^<])(==>+)", r" <==>", temp)
temp = re.sub(r"^(<==>)", r"0 \1", temp)
temp = re.sub(r'^(<==>)', r'0 \1', temp) temp = re.sub(r"(?<!\n) +(?!\n)", " ", temp) # remove double space
temp = re.sub(r'(?<!\n) +(?!\n)', ' ', temp) # remove double space
temp = temp.split("\n") temp = temp.split("\n")
for k, t in enumerate(temp): for k, t in enumerate(temp):
temp[k] = temp[k].split("<==>") temp[k] = temp[k].split("<==>")
temp[k][1] = temp[k][1].split("//") temp[k][1] = temp[k][1].split("//")
#for j, chunk in enumerate(data.iloc[i]["chunked_sentance1"].sorted(key=len, reverse=True)): # for j, chunk in enumerate(data.iloc[i]["chunked_sentance1"].sorted(key=len, reverse=True)):
for j, chunk in enumerate(sorted(data.iloc[i]["chunked_sentance1"], key=lambda x: len(x), reverse=True)): for j, chunk in enumerate(
sorted(
data.iloc[i]["chunked_sentance1"], key=lambda x: len(x), reverse=True
)
):
pattern = re.compile(chunk, re.IGNORECASE) pattern = re.compile(chunk, re.IGNORECASE)
temp[k][0] = pattern.sub(str(j+1), temp[k][0]) temp[k][0] = pattern.sub(str(j + 1), temp[k][0])
#temp[k][0] = temp[k][0].replace(chunk, str(j+1)) # temp[k][0] = temp[k][0].replace(chunk, str(j+1))
for j, chunk in enumerate(sorted(data.iloc[i]["chunked_sentance2"], key=lambda x: len(x), reverse=True)): for j, chunk in enumerate(
sorted(
data.iloc[i]["chunked_sentance2"], key=lambda x: len(x), reverse=True
)
):
pattern = re.compile(chunk, re.IGNORECASE) pattern = re.compile(chunk, re.IGNORECASE)
temp[k][1][0] = pattern.sub(str(j+1), temp[k][1][0]) temp[k][1][0] = pattern.sub(str(j + 1), temp[k][1][0])
#temp[k][1][0] = temp[k][1][0].replace(chunk, str(j+1)) # temp[k][1][0] = temp[k][1][0].replace(chunk, str(j+1))
if len(temp[k][1]) >= 3: if len(temp[k][1]) >= 3:
temp[k] = temp[k][0] + " <==> " + temp[k][1][0] + " // " + temp[k][1][1] + " // " + temp[k][1][2] temp[k] = (
temp[k][0]
+ " <==> "
+ temp[k][1][0]
+ " // "
+ temp[k][1][1]
+ " // "
+ temp[k][1][2]
)
elif len(temp[k][1]) == 2: elif len(temp[k][1]) == 2:
temp[k] = temp[k][0] + " <==> " + temp[k][1][0] + " // " + temp[k][1][1] + " // 0" temp[k] = (
temp[k] = re.sub(r'\s{2,}', ' ', temp[k]) # remove double space temp[k][0] + " <==> " + temp[k][1][0] + " // " + temp[k][1][1] + " // 0"
)
temp[k] = re.sub(r"\s{2,}", " ", temp[k]) # remove double space
responses[i] = "\n".join(temp) responses[i] = "\n".join(temp)
@ -88,13 +120,12 @@ for i, r in enumerate(responses):
print(unformatted[i]) print(unformatted[i])
# write to file # write to file
file_path = "student_fixed_format_with_training.txt" file_path = "student_fixed_format_with_training.txt"
with open(file_path, 'w') as file: with open(file_path, "w") as file:
for i, r in enumerate(responses): for i, r in enumerate(responses):
file.write("<sentence id=\"" + str(i+1) + "\" status=\"\">\n") file.write('<sentence id="' + str(i + 1) + '" status="">\n')
file.write("<alignment>\n") file.write("<alignment>\n")
file.write(r) file.write(r)
file.write("\n</alignment>\n") file.write("\n</alignment>\n")

27
Programming/NLP/gpt_alignment.py
View File

@ -1,9 +1,7 @@
""" """Get chunk alignment from chatGPT."""
Get chunk alignment from chatGPT
"""
from openai import OpenAI
import pandas as pd import pandas as pd
import processing from openai import OpenAI
thePrompt = """You are a machine designed to align chunks from 2 sentences. This means you will be taking each chunk from one sentence and comparing it to every chunk from the other sentence. Choose the chunks with the strongest relation and assign them a score. thePrompt = """You are a machine designed to align chunks from 2 sentences. This means you will be taking each chunk from one sentence and comparing it to every chunk from the other sentence. Choose the chunks with the strongest relation and assign them a score.
@ -54,28 +52,28 @@ def createGPT() -> OpenAI:
return client return client
def callApi(client:OpenAI, chunks:str): def callApi(client: OpenAI, chunks: str):
response = client.chat.completions.create( response = client.chat.completions.create(
model="gpt-3.5-turbo-1106", model="gpt-3.5-turbo-1106",
messages=[ messages=[
{"role": "user", "content": thePrompt}, {"role": "user", "content": thePrompt},
{"role": "user", "content": chunks} {"role": "user", "content": chunks},
] ],
) )
return response.choices[0].message.content.strip() return response.choices[0].message.content.strip()
def callApi_examples(client:OpenAI, examples:pd.DataFrame, alignment:str): def callApi_examples(client: OpenAI, examples: pd.DataFrame, alignment: str):
user_input = [] user_input = []
assistant_output = [] assistant_output = []
for index, row in examples.iterrows(): for index, row in examples.iterrows():
chunks = "" chunks = ""
for chunk in row["chunked_sentance1"]: for chunk in row["chunked_sentance1"]:
chunks = chunks + "[ " + chunk + " ] " chunks = chunks + "[ " + chunk + " ] "
chunks = chunks + "\n" chunks = chunks + "\n"
for chunk in row["chunked_sentance2"]: for chunk in row["chunked_sentance2"]:
chunks = chunks + "[ " + chunk + " ] " chunks = chunks + "[ " + chunk + " ] "
user_input.append(chunks) user_input.append(chunks)
assistant_output.append(row["alignment_text"]) assistant_output.append(row["alignment_text"])
@ -88,7 +86,6 @@ def callApi_examples(client:OpenAI, examples:pd.DataFrame, alignment:str):
messages.append({"role": "user", "content": alignment}) messages.append({"role": "user", "content": alignment})
response = client.chat.completions.create( response = client.chat.completions.create(
model="gpt-3.5-turbo-1106", model="gpt-3.5-turbo-1106", messages=messages
messages=messages
) )
return response.choices[0].message.content.strip() return response.choices[0].message.content.strip()

386
Programming/NLP/gpt_chunks.py
View File

@ -1,25 +1,27 @@
""" """Orders gpt to chunkify files from task."""
Orders gpt to chunkify files from task
"""
import re
import os import os
import re
from openai import OpenAI from openai import OpenAI
def sentence_loop(sentences): def sentence_loop(sentences):
""" """Processes a list of sentences, applying chunking and formatting to each.
Processes a list of sentences, applying chunking and formatting to each.
This function iterates through a list of sentences, processing each one This function iterates through a list of sentences, processing each one
using the `process_sentence` function. It collects the results into a list. using the `process_sentence` function. It collects the results into a list.
If processing fails for any sentence, the function halts and returns None. If processing fails for any sentence, the function halts and returns None.
Parameters: Parameters
----------
sentences (list of str): A list of sentences to be processed. sentences (list of str): A list of sentences to be processed.
Returns: Returns
-------
list of str: A list of processed and formatted sentences, or None if processing list of str: A list of processed and formatted sentences, or None if processing
of any sentence fails. of any sentence fails.
""" """
# Process each sentence # Process each sentence
chunked_sentences = [] chunked_sentences = []
@ -31,26 +33,27 @@ def sentence_loop(sentences):
def remove_leading_numbering(input_string): def remove_leading_numbering(input_string):
""" """Removes any leading numbering from the string.
Removes any leading numbering from the string.
This function is designed to strip off numbering This function is designed to strip off numbering
that typically appears at the beginning of a string, that typically appears at the beginning of a string,
such as '1. ', '2. ', etc. It's useful for processing strings where such numbering is irrelevant such as '1. ', '2. ', etc. It's useful for processing strings where such numbering is irrelevant
or should be omitted for formatting purposes. or should be omitted for formatting purposes.
Parameters: Parameters
----------
string (str): The string from which leading numbering should be removed. string (str): The string from which leading numbering should be removed.
Returns: Returns
-------
str: The string with leading numbering removed. str: The string with leading numbering removed.
""" """
return re.sub(r"^\d+\.\s*", "", input_string) return re.sub(r"^\d+\.\s*", "", input_string)
def process_brackets(input_string): def process_brackets(input_string):
""" """Processes a string to extract and adjust parts enclosed in square brackets.
Processes a string to extract and adjust parts enclosed in square brackets.
This function identifies and extracts all segments within square brackets from the input string. This function identifies and extracts all segments within square brackets from the input string.
It also adjusts the last part by handling any trailing punctuation, It also adjusts the last part by handling any trailing punctuation,
@ -58,11 +61,14 @@ def process_brackets(input_string):
This is particularly useful for strings This is particularly useful for strings
that contain multiple bracketed segments that need individual processing. that contain multiple bracketed segments that need individual processing.
Parameters: Parameters
----------
input_string (str): The string to be processed for bracketed segments. input_string (str): The string to be processed for bracketed segments.
Returns: Returns
-------
list of str: A list containing the extracted and adjusted bracketed segments from the string. list of str: A list containing the extracted and adjusted bracketed segments from the string.
""" """
parts = re.findall(r"\[[^\]]*\]", input_string) parts = re.findall(r"\[[^\]]*\]", input_string)
if parts: if parts:
@ -71,20 +77,22 @@ def process_brackets(input_string):
def handle_trailing_punctuation(input_string, last_part): def handle_trailing_punctuation(input_string, last_part):
""" """Handles and appends trailing punctuation to the last part of a bracketed segment.
Handles and appends trailing punctuation to the last part of a bracketed segment.
This function checks the given string for any punctuation (like '.', '!', or '?') This function checks the given string for any punctuation (like '.', '!', or '?')
immediately following the last bracketed segment. If found, it appends this punctuation immediately following the last bracketed segment. If found, it appends this punctuation
to the end of the last part inside the brackets. This is used to maintain sentence to the end of the last part inside the brackets. This is used to maintain sentence
punctuation integrity after processing bracketed segments. punctuation integrity after processing bracketed segments.
Parameters: Parameters
----------
input_string (str): The original string containing the bracketed segments. input_string (str): The original string containing the bracketed segments.
last_part (str): The last bracketed segment extracted from the string. last_part (str): The last bracketed segment extracted from the string.
Returns: Returns
-------
str: The last part with any trailing punctuation correctly appended. str: The last part with any trailing punctuation correctly appended.
""" """
trailing_punctuation = re.search(r"\]\s*([.!?])\s*$", input_string) trailing_punctuation = re.search(r"\]\s*([.!?])\s*$", input_string)
if trailing_punctuation: if trailing_punctuation:
@ -93,32 +101,36 @@ def handle_trailing_punctuation(input_string, last_part):
def process_non_bracketed_parts(input_string): def process_non_bracketed_parts(input_string):
""" """Split the input string by hyphens surrounded by optional whitespace and return
Split the input string by hyphens surrounded by optional whitespace and return
a list of non-empty parts enclosed in square brackets. a list of non-empty parts enclosed in square brackets.
Parameters: Parameters
----------
input_string (str): The input string to be processed. input_string (str): The input string to be processed.
Returns: Returns
-------
list of str: A list of non-empty parts from the input string, enclosed in square brackets. list of str: A list of non-empty parts from the input string, enclosed in square brackets.
""" """
parts = re.split(r"\s*-\s*", input_string) parts = re.split(r"\s*-\s*", input_string)
return [f"[ {part.strip()} ]" for part in parts if part.strip()] return [f"[ {part.strip()} ]" for part in parts if part.strip()]
def reformat_list(strings): def reformat_list(strings):
""" """Reformat a list of strings by processing each string element. For strings containing
Reformat a list of strings by processing each string element. For strings containing
square brackets, it calls 'process_brackets' to handle the content within the brackets, square brackets, it calls 'process_brackets' to handle the content within the brackets,
and for other strings, it calls 'process_non_bracketed_parts' to handle them. The and for other strings, it calls 'process_non_bracketed_parts' to handle them. The
resulting parts are joined into a single string with spaces. resulting parts are joined into a single string with spaces.
Parameters: Parameters
----------
strings (list of str): A list of strings to be reformatted. strings (list of str): A list of strings to be reformatted.
Returns: Returns
-------
str: A reformatted string containing the processed parts of input strings. str: A reformatted string containing the processed parts of input strings.
""" """
result = [] result = []
for string in strings: for string in strings:
@ -131,18 +143,20 @@ def reformat_list(strings):
def reformat_square_not_full(input_string): def reformat_square_not_full(input_string):
""" """Reformat the input string by splitting it into parts that are either enclosed in
Reformat the input string by splitting it into parts that are either enclosed in
square brackets or not. For parts not in brackets, it removes trailing periods, square brackets or not. For parts not in brackets, it removes trailing periods,
trims spaces, and encloses them in square brackets. For parts already in brackets, trims spaces, and encloses them in square brackets. For parts already in brackets,
it adds spaces inside the brackets. The formatted parts are then joined into a it adds spaces inside the brackets. The formatted parts are then joined into a
single string with spaces. single string with spaces.
Parameters: Parameters
----------
input_string (str): The input string to be reformatted. input_string (str): The input string to be reformatted.
Returns: Returns
-------
str: A reformatted string containing the processed parts of the input string. str: A reformatted string containing the processed parts of the input string.
""" """
# Split the string into parts that are either in brackets or not # Split the string into parts that are either in brackets or not
parts = re.split(r"(\[[^\]]*\])", input_string) parts = re.split(r"(\[[^\]]*\])", input_string)
@ -163,16 +177,18 @@ def reformat_square_not_full(input_string):
def reformat_slash(input_string): def reformat_slash(input_string):
""" """Reformat the input string by splitting it on slashes, trimming spaces from each
Reformat the input string by splitting it on slashes, trimming spaces from each
part, and enclosing each part in square brackets. The formatted parts are then part, and enclosing each part in square brackets. The formatted parts are then
joined into a single string with spaces. joined into a single string with spaces.
Parameters: Parameters
----------
input_string (str): The input string to be reformatted. input_string (str): The input string to be reformatted.
Returns: Returns
-------
str: A reformatted string containing the processed parts of the input string. str: A reformatted string containing the processed parts of the input string.
""" """
# Split the string on slashes and strip spaces # Split the string on slashes and strip spaces
parts = [part.strip() for part in input_string.split("/")] parts = [part.strip() for part in input_string.split("/")]
@ -182,16 +198,18 @@ def reformat_slash(input_string):
def reformat_pipe(input_string): def reformat_pipe(input_string):
""" """Reformat the input string by splitting it on ' | ' (pipe symbol with spaces),
Reformat the input string by splitting it on ' | ' (pipe symbol with spaces),
trimming spaces from each part, and enclosing each part in square brackets. The trimming spaces from each part, and enclosing each part in square brackets. The
formatted parts are then joined into a single string with spaces. formatted parts are then joined into a single string with spaces.
Parameters: Parameters
----------
input_string (str): The input string to be reformatted. input_string (str): The input string to be reformatted.
Returns: Returns
-------
str: A reformatted string containing the processed parts of the input string. str: A reformatted string containing the processed parts of the input string.
""" """
# Split the string on ' | ' and strip spaces # Split the string on ' | ' and strip spaces
parts = [part.strip() for part in input_string.split("|")] parts = [part.strip() for part in input_string.split("|")]
@ -201,14 +219,16 @@ def reformat_pipe(input_string):
def is_pipe_format(input_string): def is_pipe_format(input_string):
""" """Check if the input string follows the ' | ' (pipe symbol with spaces) pattern.
Check if the input string follows the ' | ' (pipe symbol with spaces) pattern.
Parameters: Parameters
----------
input_string (str): The input string to be checked. input_string (str): The input string to be checked.
Returns: Returns
-------
bool: True if the input string matches the pattern, otherwise False. bool: True if the input string matches the pattern, otherwise False.
""" """
# Regular expression to check for ' | ' pattern # Regular expression to check for ' | ' pattern
pattern = r"^(?:[^|]+\|)+[^|]+$" pattern = r"^(?:[^|]+\|)+[^|]+$"
@ -216,20 +236,22 @@ def is_pipe_format(input_string):
def is_numbered_line_format(input_string): def is_numbered_line_format(input_string):
""" """Check if the input string(input_string) follows the pattern of lines starting with
Check if the input string(input_string) follows the pattern of lines starting with
'number. word/phrase \n'. 'number. word/phrase \n'.
Parameters: Parameters
----------
input_string (str or list of str): The input string(input_string) to be checked. input_string (str or list of str): The input string(input_string) to be checked.
It can be a single string or a list of strings. It can be a single string or a list of strings.
Returns: Returns
-------
bool or list of bool: bool or list of bool:
If the input is a single string, it returns True If the input is a single string, it returns True
if it matches the pattern, otherwise False. if it matches the pattern, otherwise False.
If the input is a list of strings If the input is a list of strings
it returns a list of booleans indicating if each string matches the pattern. it returns a list of booleans indicating if each string matches the pattern.
""" """
# Regular expression to match lines starting with 'number. word/phrase \n' # Regular expression to match lines starting with 'number. word/phrase \n'
pattern = r"^(?:\d+\.\s+.+\n?)+$" pattern = r"^(?:\d+\.\s+.+\n?)+$"
@ -243,17 +265,19 @@ def is_numbered_line_format(input_string):
def reformat_ists(input_string): def reformat_ists(input_string):
""" """Reformat the input string by finding all occurrences of the (iSTS n) pattern
Reformat the input string by finding all occurrences of the (iSTS n) pattern
(where 'n' is a number), extracting the text inside the parentheses, and (where 'n' is a number), extracting the text inside the parentheses, and
enclosing each extracted part in square brackets. The formatted parts are enclosing each extracted part in square brackets. The formatted parts are
then joined into a single string with spaces. then joined into a single string with spaces.
Parameters: Parameters
----------
input_string (str): The input string to be reformatted. input_string (str): The input string to be reformatted.
Returns: Returns
-------
str: A reformatted string containing the processed parts of the input string. str: A reformatted string containing the processed parts of the input string.
""" """
# Use regular expression to find all occurrences of the iSTS pattern and # Use regular expression to find all occurrences of the iSTS pattern and
# extract the text # extract the text
@ -264,15 +288,17 @@ def reformat_ists(input_string):
def is_ists_format(input_string): def is_ists_format(input_string):
""" """Check if the input string follows the pattern of (iSTS n) followed by text, with
Check if the input string follows the pattern of (iSTS n) followed by text, with
multiple occurrences allowed, where 'n' is a number. multiple occurrences allowed, where 'n' is a number.
Parameters: Parameters
----------
input_string (str): The input string to be checked. input_string (str): The input string to be checked.
Returns: Returns
-------
bool: True if the input string matches the pattern, otherwise False. bool: True if the input string matches the pattern, otherwise False.
""" """
# Regular expression to check for the iSTS pattern # Regular expression to check for the iSTS pattern
pattern = r"^(?:\(iSTS \d+\)\s*[^)]+\s*)+$" pattern = r"^(?:\(iSTS \d+\)\s*[^)]+\s*)+$"
@ -280,17 +306,19 @@ def is_ists_format(input_string):
def reformat_with_sections(input_string): def reformat_with_sections(input_string):
""" """Reformat the input string by using regular expressions to split it into parts
Reformat the input string by using regular expressions to split it into parts
following section labels (e.g., [S1], [S2], etc.), removing empty strings, following section labels (e.g., [S1], [S2], etc.), removing empty strings,
and enclosing each part in square brackets. The formatted parts are then and enclosing each part in square brackets. The formatted parts are then
joined into a single string with spaces. joined into a single string with spaces.
Parameters: Parameters
----------
input_string (str): The input string to be reformatted. input_string (str): The input string to be reformatted.
Returns: Returns
-------
str: A reformatted string containing the processed parts of the input string. str: A reformatted string containing the processed parts of the input string.
""" """
# Use regular expression to extract text following the section labels # Use regular expression to extract text following the section labels
parts = re.split(r"\s*\[\s*S\d+\s*\]\s*", input_string) parts = re.split(r"\s*\[\s*S\d+\s*\]\s*", input_string)
@ -300,15 +328,17 @@ def reformat_with_sections(input_string):
def is_section_format(input_string): def is_section_format(input_string):
""" """Check if the input string follows the pattern of section labels (e.g., [S1], [S2], etc.)
Check if the input string follows the pattern of section labels (e.g., [S1], [S2], etc.)
followed by text, with multiple occurrences allowed. followed by text, with multiple occurrences allowed.
Parameters: Parameters
----------
input_string (str): The input string to be checked. input_string (str): The input string to be checked.
Returns: Returns
-------
bool: True if the input string matches the pattern, otherwise False. bool: True if the input string matches the pattern, otherwise False.
""" """
# Regular expression to check for the section label pattern # Regular expression to check for the section label pattern
pattern = r"^(?:\s*\[\s*S\d+\s*\]\s*.+)+$" pattern = r"^(?:\s*\[\s*S\d+\s*\]\s*.+)+$"
@ -316,34 +346,37 @@ def is_section_format(input_string):
def is_chunk_format(input_string): def is_chunk_format(input_string):
""" """Check if the input string follows the pattern of 'Chunk X: text' or '[Chunk X] text'
Check if the input string follows the pattern of 'Chunk X: text' or '[Chunk X] text'
patterns, where 'X' is a number, with multiple occurrences allowed. patterns, where 'X' is a number, with multiple occurrences allowed.
Parameters: Parameters
----------
input_string (str): The input string to be checked. input_string (str): The input string to be checked.
Returns: Returns
-------
bool: True if the input string matches the pattern, otherwise False. bool: True if the input string matches the pattern, otherwise False.
""" """
# Further optimized regular expression # Further optimized regular expression
pattern = r"^(Chunk \d+:|\[\s*Chunk \d+\s*\]).*?(?:\s+(?:Chunk \d+:|\[\s*Chunk \d+\s*\]).*?)*$" pattern = r"^(Chunk \d+:|\[\s*Chunk \d+\s*\]).*?(?:\s+(?:Chunk \d+:|\[\s*Chunk \d+\s*\]).*?)*$"
return bool(re.match(pattern, input_string)) return bool(re.match(pattern, input_string))
def reformat_chunks(input_string): def reformat_chunks(input_string):
""" """Reformat the input string by finding all occurrences of both
Reformat the input string by finding all occurrences of both
'Chunk X: text' and '[Chunk X] text' 'Chunk X: text' and '[Chunk X] text'
patterns, extracting the text within them, and enclosing each extracted part in square brackets. patterns, extracting the text within them, and enclosing each extracted part in square brackets.
The formatted parts are then joined into a single string with spaces. The formatted parts are then joined into a single string with spaces.
Parameters: Parameters
----------
input_string (str): The input string to be reformatted. input_string (str): The input string to be reformatted.
Returns: Returns
-------
str: A reformatted string containing the processed parts of the input string. str: A reformatted string containing the processed parts of the input string.
""" """
# Find all occurrences of both chunk patterns and extract the text # Find all occurrences of both chunk patterns and extract the text
parts = re.findall( parts = re.findall(
@ -360,16 +393,18 @@ def reformat_chunks(input_string):
def reformat_ists_markers(input_string): def reformat_ists_markers(input_string):
""" """Reformat the input string by splitting it at '[ iSTS ]' markers, removing empty strings,
Reformat the input string by splitting it at '[ iSTS ]' markers, removing empty strings,
and enclosing each part in square brackets. The formatted parts are then joined into and enclosing each part in square brackets. The formatted parts are then joined into
a single string with spaces. a single string with spaces.
Parameters: Parameters
----------
input_string (str): The input string to be reformatted. input_string (str): The input string to be reformatted.
Returns: Returns
-------
str: A reformatted string containing the processed parts of the input string. str: A reformatted string containing the processed parts of the input string.
""" """
# Split the string at '[ iSTS ]', remove empty strings, and enclose each # Split the string at '[ iSTS ]', remove empty strings, and enclose each
# part in square brackets # part in square brackets
@ -383,15 +418,17 @@ def reformat_ists_markers(input_string):
def is_ists_marker_format(input_string): def is_ists_marker_format(input_string):
""" """Check if the input string follows the pattern with '[ iSTS ]' markers followed by text,
Check if the input string follows the pattern with '[ iSTS ]' markers followed by text,
with multiple occurrences allowed. with multiple occurrences allowed.
Parameters: Parameters
----------
input_string (str): The input string to be checked. input_string (str): The input string to be checked.
Returns: Returns
-------
bool: True if the input string matches the pattern, otherwise False. bool: True if the input string matches the pattern, otherwise False.
""" """
# Regular expression to check for the pattern with '[ iSTS ]' # Regular expression to check for the pattern with '[ iSTS ]'
pattern = r"^(?:\[\s*iSTS\s*\].+)+$" pattern = r"^(?:\[\s*iSTS\s*\].+)+$"
@ -399,16 +436,18 @@ def is_ists_marker_format(input_string):
def reformat_with_dashes(input_string): def reformat_with_dashes(input_string):
""" """Reformat the input string by splitting it on ' - ' (dash with spaces),
Reformat the input string by splitting it on ' - ' (dash with spaces),
removing empty strings, and enclosing each part in square brackets. removing empty strings, and enclosing each part in square brackets.
The formatted parts are then joined into a single string with spaces. The formatted parts are then joined into a single string with spaces.
Parameters: Parameters
----------
input_string (str): The input string to be reformatted. input_string (str): The input string to be reformatted.
Returns: Returns
-------
str: A reformatted string containing the processed parts of the input string. str: A reformatted string containing the processed parts of the input string.
""" """
# Split the string on ' - ', remove empty strings, and enclose each part # Split the string on ' - ', remove empty strings, and enclose each part
# in square brackets # in square brackets
@ -418,15 +457,17 @@ def reformat_with_dashes(input_string):
def is_dash_format(input_string): def is_dash_format(input_string):
""" """Check if the input string follows the pattern with ' - ' (dash with spaces)
Check if the input string follows the pattern with ' - ' (dash with spaces)
separating non-empty parts. separating non-empty parts.
Parameters: Parameters
----------
input_string (str): The input string to be checked. input_string (str): The input string to be checked.
Returns: Returns
-------
bool: True if the input string matches the pattern, otherwise False. bool: True if the input string matches the pattern, otherwise False.
""" """
# Regular expression to check for the pattern with ' - ' # Regular expression to check for the pattern with ' - '
pattern = r"^[^-]+(?:\s*-\s*[^-]+)+$" pattern = r"^[^-]+(?:\s*-\s*[^-]+)+$"
@ -434,19 +475,21 @@ def is_dash_format(input_string):
def preliminary_reformat(input_string): def preliminary_reformat(input_string):
""" """Remove common prefixes like
Remove common prefixes like
'iSTS chunks:' 'iSTS chunks:'
and and
'Here are the iSTS chunks for the given sentence:' 'Here are the iSTS chunks for the given sentence:'
from the input string. from the input string.
This function is intended as a preliminary step before further reformatting. This function is intended as a preliminary step before further reformatting.
Parameters: Parameters
----------
input_string (str): The input string to be reformatted. input_string (str): The input string to be reformatted.
Returns: Returns
-------
str: The input string with common prefixes stripped. str: The input string with common prefixes stripped.
""" """
input_string = input_string.strip("iSTS chunks:") input_string = input_string.strip("iSTS chunks:")
input_string = input_string.strip( input_string = input_string.strip(
@ -456,17 +499,19 @@ def preliminary_reformat(input_string):
def reformat_brackets_and_text(input_string): def reformat_brackets_and_text(input_string):
""" """Reformat the input string by removing trailing dots, splitting it into parts that are either
Reformat the input string by removing trailing dots, splitting it into parts that are either
enclosed in square brackets or not, removing commas and trimming spaces for non-bracket parts, enclosed in square brackets or not, removing commas and trimming spaces for non-bracket parts,
and enclosing non-bracket parts in square brackets. The formatted parts are then joined into and enclosing non-bracket parts in square brackets. The formatted parts are then joined into
a single string with spaces. a single string with spaces.
Parameters: Parameters
----------
input_string (str): The input string to be reformatted. input_string (str): The input string to be reformatted.
Returns: Returns
-------
str: A reformatted string containing the processed parts of the input string. str: A reformatted string containing the processed parts of the input string.
""" """
# Remove the trailing dot if present # Remove the trailing dot if present
input_string = re.sub(r"\.$", "", input_string) input_string = re.sub(r"\.$", "", input_string)
@ -490,16 +535,18 @@ def reformat_brackets_and_text(input_string):
def is_brackets_and_text_format(input_string): def is_brackets_and_text_format(input_string):
""" """Check if the input string follows the pattern of parts
Check if the input string follows the pattern of parts
that are either enclosed in square brackets that are either enclosed in square brackets
or not, with optional commas and spaces between the parts. or not, with optional commas and spaces between the parts.
Parameters: Parameters
----------
input_string (str): The input string to be checked. input_string (str): The input string to be checked.
Returns: Returns
-------
bool: True if the input string matches the pattern, otherwise False. bool: True if the input string matches the pattern, otherwise False.
""" """
# Regular expression to check the pattern # Regular expression to check the pattern
pattern = r"^(\[[^\]]*\]|\s*[^[\]]+\s*)(,\s*|\s+|$)+$" pattern = r"^(\[[^\]]*\]|\s*[^[\]]+\s*)(,\s*|\s+|$)+$"
@ -507,18 +554,20 @@ def is_brackets_and_text_format(input_string):
def is_correct_format(input_string): def is_correct_format(input_string):
""" """Checks if a given string matches a specific format of sequences enclosed in square brackets.
Checks if a given string matches a specific format of sequences enclosed in square brackets.
This function uses a regular expression to determine if the input string adheres to a format This function uses a regular expression to determine if the input string adheres to a format
where sequences of text are enclosed within square brackets. Each sequence must be separated where sequences of text are enclosed within square brackets. Each sequence must be separated
by spaces, and the entire string must only consist of these bracketed sequences. by spaces, and the entire string must only consist of these bracketed sequences.
Parameters: Parameters
----------
input_string (str): The string to be checked against the format. input_string (str): The string to be checked against the format.
Returns: Returns
-------
bool: True if the string matches the format, False otherwise. bool: True if the string matches the format, False otherwise.
""" """
# Regular expression to match sequences of '[ some words ]' # Regular expression to match sequences of '[ some words ]'
pattern = r"^(\[\s*[^]]+\s*\]\s*)+$" pattern = r"^(\[\s*[^]]+\s*\]\s*)+$"
@ -528,8 +577,7 @@ def is_correct_format(input_string):
def check_format(input_data): def check_format(input_data):
""" """Checks if the input data (input_data or list of strings) matches a specific format.
Checks if the input data (input_data or list of strings) matches a specific format.
This function determines if the input data, This function determines if the input data,
which can be either a single string or a list of strings, which can be either a single string or a list of strings,
@ -537,16 +585,19 @@ def check_format(input_data):
The format check is performed by the 'is_correct_format' function. The format check is performed by the 'is_correct_format' function.
For a list, the check is applied to each string in the list. For a list, the check is applied to each string in the list.
Parameters: Parameters
----------
input_data (str or list of str): input_data (str or list of str):
The input data to be checked. The input data to be checked.
Can be a single string or a list of strings. Can be a single string or a list of strings.
Returns: Returns
-------
bool or list of bool: If input_data is a string, bool or list of bool: If input_data is a string,
returns True if it matches the format, False otherwise. returns True if it matches the format, False otherwise.
If input_data is a list, returns a list of booleans corresponding to each string. If input_data is a list, returns a list of booleans corresponding to each string.
Returns None if input_data is neither a string nor a list. Returns None if input_data is neither a string nor a list.
""" """
# Check if the input is a string, and if so, process it directly # Check if the input is a string, and if so, process it directly
if isinstance(input_data, str): if isinstance(input_data, str):
@ -559,27 +610,29 @@ def check_format(input_data):
# Return None or raise an error if the input is neither a string nor a list # Return None or raise an error if the input is neither a string nor a list
return None return None
def check_and_reformat(input_string, check_func, format_func): def check_and_reformat(input_string, check_func, format_func):
""" """Checks what type of string got inputed and formats it accordingly."""
Checks what type of string got inputed and formats it accordingly
"""
if check_func(input_string): if check_func(input_string):
print(f"formatted by {format_func.__name__}", input_string) print(f"formatted by {format_func.__name__}", input_string)
return format_func(input_string) return format_func(input_string)
return None return None
def reformat(input_string): def reformat(input_string):
""" """Reformat the input string by applying a series of format checks and corresponding
Reformat the input string by applying a series of format checks and corresponding
reformatting functions. This function attempts to identify the format of the input reformatting functions. This function attempts to identify the format of the input
string and apply the appropriate reformatting. string and apply the appropriate reformatting.
Parameters: Parameters
----------
input_string (str): The input string to be reformatted. input_string (str): The input string to be reformatted.
Returns: Returns
-------
str or bool: The reformatted string if a suitable format is recognized, or False if the format str or bool: The reformatted string if a suitable format is recognized, or False if the format
is not recognized and cannot be reformatted. is not recognized and cannot be reformatted.
""" """
input_string = preliminary_reformat(input_string) input_string = preliminary_reformat(input_string)
if check_format(input_string): if check_format(input_string):
@ -608,15 +661,17 @@ def reformat(input_string):
def remove_empty_brackets(input_string): def remove_empty_brackets(input_string):
""" """Remove empty brackets (with or without spaces) from the input string and
Remove empty brackets (with or without spaces) from the input string and
eliminate any extra spaces that might be left after removing brackets. eliminate any extra spaces that might be left after removing brackets.
Parameters: Parameters
----------
input_string (str): The input string from which empty brackets should be removed. input_string (str): The input string from which empty brackets should be removed.
Returns: Returns
-------
str: The input string with empty brackets removed and extra spaces eliminated. str: The input string with empty brackets removed and extra spaces eliminated.
""" """
# Regular expression to match empty brackets (with any number of spaces) # Regular expression to match empty brackets (with any number of spaces)
pattern = r"\[\s*\]" pattern = r"\[\s*\]"
@ -631,30 +686,34 @@ def remove_empty_brackets(input_string):
def reformat_brackets(input_string): def reformat_brackets(input_string):
""" """Reformat the input string by enclosing text inside square brackets.
Reformat the input string by enclosing text inside square brackets.
This function searches for text enclosed in square brackets (with or without spaces) This function searches for text enclosed in square brackets (with or without spaces)
and ensures that the text inside the brackets is surrounded by spaces. and ensures that the text inside the brackets is surrounded by spaces.
Parameters: Parameters
----------
input_string (str): The input string to be reformatted. input_string (str): The input string to be reformatted.
Returns: Returns
-------
str: A reformatted string with text inside square brackets and spaces around it. str: A reformatted string with text inside square brackets and spaces around it.
""" """
formatted_string = re.sub(r"\[\s*(.*?)\s*\]", r"[ \1 ]", input_string) formatted_string = re.sub(r"\[\s*(.*?)\s*\]", r"[ \1 ]", input_string)
return formatted_string return formatted_string
def generate_instruction(sentence): def generate_instruction(sentence):
""" """Generate an instruction for dividing a given sentence into iSTS chunks.
Generate an instruction for dividing a given sentence into iSTS chunks.
Parameters: Parameters
----------
sentence (str): The sentence to be divided into chunks. sentence (str): The sentence to be divided into chunks.
Returns: Returns
-------
str: A formatted instruction with the provided sentence. str: A formatted instruction with the provided sentence.
""" """
return ( return (
"Please divide the following sentence into iSTS chunks." "Please divide the following sentence into iSTS chunks."
@ -665,15 +724,17 @@ def generate_instruction(sentence):
def call_api(instruction): def call_api(instruction):
""" """Call the API using the provided instruction to generate a response.
Call the API using the provided instruction to generate a response.
Parameters: Parameters
----------
client (OpenAI.Client): An OpenAI API client object. client (OpenAI.Client): An OpenAI API client object.
instruction (str): The instruction to provide to the API. instruction (str): The instruction to provide to the API.
Returns: Returns
-------
dict: The response generated by the API. dict: The response generated by the API.
""" """
return client.chat.completions.create( return client.chat.completions.create(
messages=[ messages=[
@ -687,29 +748,33 @@ def call_api(instruction):
def process_response(response): def process_response(response):
""" """Process the response received from the API and extract the chunked sentence.
Process the response received from the API and extract the chunked sentence.
Parameters: Parameters
----------
response (dict): The response generated by the API. response (dict): The response generated by the API.
Returns: Returns
-------
str: The chunked sentence extracted from the response. str: The chunked sentence extracted from the response.
""" """
chunked_sentence = response.choices[0].message.content.strip() chunked_sentence = response.choices[0].message.content.strip()
return chunked_sentence return chunked_sentence
def check_and_format_sentence(chunked_sentence): def check_and_format_sentence(chunked_sentence):
""" """Check and format a chunked sentence by applying a series of checks and reformatting.
Check and format a chunked sentence by applying a series of checks and reformatting.
Parameters: Parameters
----------
chunked_sentence (str): The chunked sentence to be checked and formatted. chunked_sentence (str): The chunked sentence to be checked and formatted.
Returns: Returns
-------
str or bool: The formatted chunked sentence if it passes all checks, str or bool: The formatted chunked sentence if it passes all checks,
or False if there are errors. or False if there are errors.
""" """
reformated = reformat(chunked_sentence) reformated = reformat(chunked_sentence)
if not reformated: if not reformated:
@ -723,15 +788,17 @@ def check_and_format_sentence(chunked_sentence):
def finalize_formatting(reformatted): def finalize_formatting(reformatted):
""" """Finalize the formatting of a reformatted string by removing empty brackets and ensuring
Finalize the formatting of a reformatted string by removing empty brackets and ensuring
that text inside brackets is surrounded by spaces. that text inside brackets is surrounded by spaces.
Parameters: Parameters
----------
reformatted (str): The reformatted string to be finalized. reformatted (str): The reformatted string to be finalized.
Returns: Returns
-------
str: The finalized and formatted string. str: The finalized and formatted string.
""" """
reformatted = remove_empty_brackets(reformatted) reformatted = remove_empty_brackets(reformatted)
reformatted = reformat_brackets(reformatted) reformatted = reformat_brackets(reformatted)
@ -739,18 +806,20 @@ def finalize_formatting(reformatted):
def process_sentence(sentence, chunked_sentences): def process_sentence(sentence, chunked_sentences):
""" """Process a sentence by generating iSTS chunks and
Process a sentence by generating iSTS chunks and
adding them to the provided list of chunked sentences. adding them to the provided list of chunked sentences.
Parameters: Parameters
----------
sentence (str): The input sentence to be processed. sentence (str): The input sentence to be processed.
chunked_sentences (list): A list of previously chunked sentences to which the processed sentence chunked_sentences (list): A list of previously chunked sentences to which the processed sentence
will be appended. will be appended.
client (OpenAI.Client): An OpenAI API client object for calling the API. client (OpenAI.Client): An OpenAI API client object for calling the API.
Returns: Returns
-------
list or bool: The updated list of chunked sentences if successful, or False if there are errors. list or bool: The updated list of chunked sentences if successful, or False if there are errors.
""" """
instruction = generate_instruction(sentence) instruction = generate_instruction(sentence)
response = call_api(instruction) response = call_api(instruction)
@ -764,19 +833,21 @@ def process_sentence(sentence, chunked_sentences):
def chunk_sentences(input_file, output): def chunk_sentences(input_file, output):
""" """Read sentences from a file, process each sentence to generate iSTS chunks,
Read sentences from a file, process each sentence to generate iSTS chunks,
and write the chunked sentences to an output file. and write the chunked sentences to an output file.
Parameters: Parameters
----------
file_path (str): The path to the input file containing sentences to be processed. file_path (str): The path to the input file containing sentences to be processed.
output_path (str): The path to the output file where chunked sentences will be written. output_path (str): The path to the output file where chunked sentences will be written.
Returns: Returns
-------
None None
""" """
# Read the sentences from the file # Read the sentences from the file
with open(input_file, "r", encoding="utf-8") as file: with open(input_file, encoding="utf-8") as file:
sentences = file.readlines() sentences = file.readlines()
# Process each sentence # Process each sentence
@ -785,8 +856,7 @@ def chunk_sentences(input_file, output):
# Write the chunked sentences to a new file # Write the chunked sentences to a new file
with open(output, "w", encoding="utf-8") as output_file: with open(output, "w", encoding="utf-8") as output_file:
for sentence in chunked_sentences: output_file.writelines(sentence + "\n" for sentence in chunked_sentences)
output_file.write(sentence + "\n")
# Usage # Usage
@ -797,7 +867,7 @@ input_files = [
"test_goldStandard/images/STSint.testinput.images.sent1.txt", "test_goldStandard/images/STSint.testinput.images.sent1.txt",
"test_goldStandard/images/STSint.testinput.images.sent2.txt", "test_goldStandard/images/STSint.testinput.images.sent2.txt",
"test_goldStandard/headlines/STSint.testinput.headlines.sent1.txt", "test_goldStandard/headlines/STSint.testinput.headlines.sent1.txt",
"test_goldStandard/headlines/STSint.testinput.headlines.sent2.txt" "test_goldStandard/headlines/STSint.testinput.headlines.sent2.txt",
] ]
output_files = [ output_files = [
@ -806,11 +876,11 @@ output_files = [
"test_goldStandard/images/images-chunks-gpt-one.txt", "test_goldStandard/images/images-chunks-gpt-one.txt",
"test_goldStandard/images/images-chunks-gpt-two.txt", "test_goldStandard/images/images-chunks-gpt-two.txt",
"test_goldStandard/headlines/headlines-chunks-gpt-one.txt", "test_goldStandard/headlines/headlines-chunks-gpt-one.txt",
"test_goldStandard/headlines/headlines-chunks-gpt-two.txt" "test_goldStandard/headlines/headlines-chunks-gpt-two.txt",
] ]
# Change me to os.environ['API_KEY'] # Change me to os.environ['API_KEY']
client = OpenAI(api_key=os.environ['API_KEY']) client = OpenAI(api_key=os.environ["API_KEY"])
for input_path, output_path in zip(input_files, output_files): for input_path, output_path in zip(input_files, output_files):
chunk_sentences(input_path, output_path) chunk_sentences(input_path, output_path)

220
Programming/NLP/processing.py
View File

@ -1,154 +1,146 @@
import pandas as pd
import numpy as np
from lxml import etree
import re import re
import numpy as np
import pandas as pd
from lxml import etree
def load_sentences(senteance1_path: str, sentance2_path: str) -> pd.DataFrame: def load_sentences(senteance1_path: str, sentance2_path: str) -> pd.DataFrame:
""" """Loads the sentences from the given paths and outputs in a 2 columns dataframe."""
Loads the sentences from the given paths and outputs in a 2 columns dataframe sentance1 = pd.read_csv(senteance1_path, dtype=str, delimiter="}", header=None)
""" sentance1.columns = ["sentance1"]
sentance1 = pd.read_csv(senteance1_path,
dtype=str,
delimiter="}",
header=None)
sentance1.columns = ["sentance1"]
sentance2 = pd.read_csv(sentance2_path, sentance2 = pd.read_csv(sentance2_path, dtype=str, delimiter="}", header=None)
dtype=str, sentance2.columns = ["sentance2"]
delimiter="}",
header=None)
sentance2.columns = ["sentance2"]
sentances = pd.concat([sentance1, sentance2], axis=1) sentances = pd.concat([sentance1, sentance2], axis=1)
return sentances return sentances
def chunk2list(chunks: str) -> list: def chunk2list(chunks: str) -> list:
""" """Takes str that is all chunks from a chunked sentance and returns a list of all the chunks as seperate items."""
Takes str that is all chunks from a chunked sentance and returns a list of all the chunks as seperate items chunks = chunks.replace("[", "")
""" chunks = chunks.replace("]", "")
chunks = chunks.replace('[', '') chunks = chunks.replace(" ", "|")
chunks = chunks.replace(']', '') split = chunks.split("|")
chunks = chunks.replace(' ', '|')
split = chunks.split('|')
split = [re.sub(r'^\s+|\s+$', '', s) for s in split] # remove spaces at the beggining and end of chunks split = [
split = [re.sub(r'[^\w\s]', '', s) for s in split] # remove punctuation re.sub(r"^\s+|\s+$", "", s) for s in split
return split ] # remove spaces at the beggining and end of chunks
split = [re.sub(r"[^\w\s]", "", s) for s in split] # remove punctuation
return split
def load_chunked(chunked_path1: str, chunked_path2: str) -> pd.DataFrame: def load_chunked(chunked_path1: str, chunked_path2: str) -> pd.DataFrame:
""" """Loads chunked sentances in [ chunk1 ] [ chunk2 ] format into dataframe with lists of chunks."""
Loads chunked sentances in [ chunk1 ] [ chunk2 ] format into dataframe with lists of chunks chunked_sentance1 = pd.read_csv(
""" chunked_path1, dtype=str, delimiter="}", header=None
chunked_sentance1 = pd.read_csv(chunked_path1, )
dtype=str, chunked_sentance1.columns = ["chunked_sentance1"]
delimiter="}",
header=None)
chunked_sentance1.columns = ["chunked_sentance1"]
chunked_sentance2 = pd.read_csv(chunked_path2, chunked_sentance2 = pd.read_csv(
dtype=str, chunked_path2, dtype=str, delimiter="}", header=None
delimiter="}", )
header=None) chunked_sentance2.columns = ["chunked_sentance2"]
chunked_sentance2.columns = ["chunked_sentance2"]
headlines_chunked = pd.concat([chunked_sentance1, chunked_sentance2], axis=1) headlines_chunked = pd.concat([chunked_sentance1, chunked_sentance2], axis=1)
# convert chunks from str to list # convert chunks from str to list
headlines_chunked['chunked_sentance1'] = headlines_chunked[ headlines_chunked["chunked_sentance1"] = headlines_chunked[
'chunked_sentance1'].apply(chunk2list) "chunked_sentance1"
headlines_chunked['chunked_sentance2'] = headlines_chunked[ ].apply(chunk2list)
'chunked_sentance2'].apply(chunk2list) headlines_chunked["chunked_sentance2"] = headlines_chunked[
"chunked_sentance2"
].apply(chunk2list)
return headlines_chunked return headlines_chunked
def return_characteers(cell: str) -> str: def return_characteers(cell: str) -> str:
""" """Converts the alignment data to restore the <==> and & tokens."""
converts the alignment data to restore the <==> and & tokens cell = cell.replace("ARROWS_PLACEHOLDER", "<==>")
""" cell = cell.replace("AMPERSAND_PLACEHOLDER", "&")
cell = cell.replace('ARROWS_PLACEHOLDER', '<==>') return cell
cell = cell.replace('AMPERSAND_PLACEHOLDER', '&')
return cell
def load_alignment(alignment_path: str) -> pd.DataFrame: def load_alignment(alignment_path: str) -> pd.DataFrame:
""" """Loads the alignment file. Parses only the <alignment> tag and puts the data into a dataframe."""
Loads the alignment file. Parses only the <alignment> tag and puts the data into a dataframe with open(alignment_path) as file:
""" file_content = file.read()
with open(alignment_path, 'r') as file:
file_content = file.read()
# <==> and & break xml loaders so it needs to be replaces with something else # <==> and & break xml loaders so it needs to be replaces with something else
modified_content = file_content.replace('<==>', 'ARROWS_PLACEHOLDER').replace('&', 'AMPERSAND_PLACEHOLDER') modified_content = file_content.replace("<==>", "ARROWS_PLACEHOLDER").replace(
# it also needs a root wrapped to function properly "&", "AMPERSAND_PLACEHOLDER"
modified_content = f'<root>{modified_content}</root>' )
# it also needs a root wrapped to function properly
modified_content = f"<root>{modified_content}</root>"
modified_file_path = 'temp.wa' modified_file_path = "temp.wa"
with open(modified_file_path, 'w') as modified_file: with open(modified_file_path, "w") as modified_file:
modified_file.write(modified_content) modified_file.write(modified_content)
# Parse the modified file using ElementTree # Parse the modified file using ElementTree
tree = etree.parse(modified_file_path) tree = etree.parse(modified_file_path)
root = tree.getroot() root = tree.getroot()
# get ansewrs # get ansewrs
alignments_data = [] alignments_data = []
for alignment in root.xpath('//alignment'): for alignment in root.xpath("//alignment"):
# Extract relevant information from the alignment element # Extract relevant information from the alignment element
data = { data = {
'sentence_id': alignment.xpath('ancestor::sentence/@id')[0], "sentence_id": alignment.xpath("ancestor::sentence/@id")[0],
'alignment_text': alignment.text "alignment_text": alignment.text,
} }
alignments_data.append(data) alignments_data.append(data)
y = pd.DataFrame(alignments_data) y = pd.DataFrame(alignments_data)
y = y.drop(columns=["sentence_id"]) y = y.drop(columns=["sentence_id"])
y["alignment_text"] = y["alignment_text"].apply(return_characteers) y["alignment_text"] = y["alignment_text"].apply(return_characteers)
return y return y
def prettyprint(element, **kwargs): def prettyprint(element, **kwargs):
xml = etree.tostring(element, pretty_print=True, **kwargs) xml = etree.tostring(element, pretty_print=True, **kwargs)
print(xml.decode(), end='') print(xml.decode(), end="")
def test_XML(): def test_XML():
# test out the format # test out the format
print(alignments_data[0]["alignment_text"]) print(alignments_data[0]["alignment_text"])
def generate_train_test_split(xy: pd.DataFrame): def generate_train_test_split(xy: pd.DataFrame):
""" """Generates a train, validate, test split of the given dataframes in a 60% 20% 20% ratio."""
Generates a train, validate, test split of the given dataframes in a 60% 20% 20% ratio # data = pd.merge(x, y, left_index=True, right_index=True)
""" data = xy
#data = pd.merge(x, y, left_index=True, right_index=True) train, test = np.split(
data = xy data.sample(frac=1, random_state=42), [int(0.03 * len(data))]
train, test = np.split(data.sample(frac=1, random_state=42), [int(.03*len(data))]) )
return train, test return train, test
def generate_alignment_format(dataFrame:pd.DataFrame, id:int) -> str:
output = "seq1:\n"
chunks1 = dataFrame["chunked_sentance1"][id]
for i, chunk in enumerate(chunks1):
output = output + str(i+1) + ") " + str(chunk) + "\n"
output = output + "\nseq2:\n"
chunks2 = dataFrame["chunked_sentance2"][id]
for i, chunk in enumerate(chunks2):
output = output + str(i+1) + ") " + str(chunk) + "\n"
return output
def get_chunks_as_text(data:pd.DataFrame) -> str: def generate_alignment_format(dataFrame: pd.DataFrame, id: int) -> str:
output = [] output = "seq1:\n"
for index, row in data.iterrows(): chunks1 = dataFrame["chunked_sentance1"][id]
chunks = "" for i, chunk in enumerate(chunks1):
for chunk in row["chunked_sentance1"]: output = output + str(i + 1) + ") " + str(chunk) + "\n"
chunks = chunks + "[ " + chunk + " ] " output = output + "\nseq2:\n"
chunks = chunks + "\n" chunks2 = dataFrame["chunked_sentance2"][id]
for chunk in row["chunked_sentance2"]: for i, chunk in enumerate(chunks2):
chunks = chunks + "[ " + chunk + " ] " output = output + str(i + 1) + ") " + str(chunk) + "\n"
output.append(chunks) return output
return output, data.index
def get_chunks_as_text(data: pd.DataFrame) -> str:
output = []
for index, row in data.iterrows():
chunks = ""
for chunk in row["chunked_sentance1"]:
chunks = chunks + "[ " + chunk + " ] "
chunks = chunks + "\n"
for chunk in row["chunked_sentance2"]:
chunks = chunks + "[ " + chunk + " ] "
output.append(chunks)
return output, data.index

4
Programming/NLP/rating.py
View File

@ -1,5 +1,5 @@
from openai import OpenAI from openai import OpenAI
def rate_similarity(c1:list, c2:list) -> str: def rate_similarity(c1: list, c2: list) -> str:
client = OpenAI(api_key='REDACTED_OPENAI_API_KEY') client = OpenAI(api_key="REDACTED_OPENAI_API_KEY")

28
Programming/NLP/test_goldStandard/chunks.py
View File

@ -1,5 +1,6 @@
from openai import OpenAI from openai import OpenAI
def sentence_loop(sentences): def sentence_loop(sentences):
# Process each sentence # Process each sentence
chunked_sentences = [] chunked_sentences = []
@ -7,12 +8,16 @@ def sentence_loop(sentences):
chunked_sentences = process_sentence(sentence, chunked_sentences) chunked_sentences = process_sentence(sentence, chunked_sentences)
return chunked_sentences return chunked_sentences
def process_sentence(sentence, chunked_sentences): def process_sentence(sentence, chunked_sentences):
response = client.chat.completions.create( response = client.chat.completions.create(
messages=[{ messages=[
"role": "user", {
"content": "Please divide the following sentence into iSTS chunks. Format the response as: [[chunk 1]] [[chunk 2]] ... Ensure each chunk is enclosed in double square brackets and separated by a space. If you cannot correctly chunkify the sentence, return an error message in the format: 'ERROR: Cannot chunkify sentence: <sentence>'. Here is the sentence: " + sentence, "role": "user",
}], "content": "Please divide the following sentence into iSTS chunks. Format the response as: [[chunk 1]] [[chunk 2]] ... Ensure each chunk is enclosed in double square brackets and separated by a space. If you cannot correctly chunkify the sentence, return an error message in the format: 'ERROR: Cannot chunkify sentence: <sentence>'. Here is the sentence: "
+ sentence,
}
],
model="gpt-3.5-turbo", model="gpt-3.5-turbo",
) )
@ -21,24 +26,25 @@ def process_sentence(sentence, chunked_sentences):
chunked_sentences.append(chunked_sentence) chunked_sentences.append(chunked_sentence)
return chunked_sentences return chunked_sentences
def chunk_sentences(file_path, output_path, api_key): def chunk_sentences(file_path, output_path, api_key):
# Read the sentences from the file # Read the sentences from the file
with open(file_path, 'r') as file: with open(file_path) as file:
sentences = file.readlines() sentences = file.readlines()
# Process each sentence # Process each sentence
chunk_sentences = sentence_loop(sentences) chunk_sentences = sentence_loop(sentences)
print(chunk_sentences) print(chunk_sentences)
# Write the chunked sentences to a new file # Write the chunked sentences to a new file
with open(output_path, 'w') as output_file: with open(output_path, "w") as output_file:
for sentence in chunk_sentences: output_file.writelines(sentence + "\n" for sentence in chunk_sentences)
output_file.write(sentence + '\n')
# Usage # Usage
file_path = 'student/STSint.testinput.answers-students.sent2.txt' file_path = "student/STSint.testinput.answers-students.sent2.txt"
output_path = 'chunks_gpt_two.txt' output_path = "chunks_gpt_two.txt"
# Replace me with os.environ['API_KEY'] # Replace me with os.environ['API_KEY']
api_key = 'REDACTED_OPENAI_API_KEY' api_key = "REDACTED_OPENAI_API_KEY"
client = OpenAI(api_key=api_key) client = OpenAI(api_key=api_key)
chunk_sentences(file_path, output_path, api_key) chunk_sentences(file_path, output_path, api_key)

949
Programming/PBAD/data.ipynb
View File

File diff suppressed because it is too large Load Diff

505
Programming/PBAD/main.ipynb
View File

@ -18,139 +18,140 @@
} }
], ],
"source": [ "source": [
"import pandas as pd\n",
"from statistics import median\n", "from statistics import median\n",
"from sklearn.model_selection import train_test_split\n", "\n",
"from sklearn.feature_selection import RFE\n",
"from sklearn.tree import DecisionTreeRegressor, plot_tree\n",
"import numpy as np\n",
"from sklearn.metrics import mean_squared_error\n",
"import matplotlib.pyplot as plt\n", "import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import pandas as pd\n",
"import seaborn as sns\n", "import seaborn as sns\n",
"from sklearn.feature_selection import RFE\n",
"from sklearn.metrics import mean_squared_error\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.tree import DecisionTreeRegressor\n",
"\n", "\n",
"feature_names = [\n", "feature_names = [\n",
" 'Powierzchnia', # 1\n", " \"Powierzchnia\", # 1\n",
" 'Wynagrodzenie_ogolem', # 2\n", " \"Wynagrodzenie_ogolem\", # 2\n",
" 'Wynagrodzenie_w_relacji_do_sredniej', # 3\n", " \"Wynagrodzenie_w_relacji_do_sredniej\", # 3\n",
" 'Dochody_podatek_lesny', # 4\n", " \"Dochody_podatek_lesny\", # 4\n",
" 'Dochody_podatek_PCC', # 5\n", " \"Dochody_podatek_PCC\", # 5\n",
" 'Dochody_podatek_od_dzialalnosci_gospodarczej', # 6\n", " \"Dochody_podatek_od_dzialalnosci_gospodarczej\", # 6\n",
" 'Dochody_podatek_od_nieruchomosci', # 7\n", " \"Dochody_podatek_od_nieruchomosci\", # 7\n",
" 'Dochody_podatek_od_spadkow', # 8\n", " \"Dochody_podatek_od_spadkow\", # 8\n",
" 'Dochody_podatek_od_srodkow_transportowych', # 9\n", " \"Dochody_podatek_od_srodkow_transportowych\", # 9\n",
" 'Dochody_podatek_rolny', # 10\n", " \"Dochody_podatek_rolny\", # 10\n",
" 'Dochody_podatek_odrebne_ustawy', # 11\n", " \"Dochody_podatek_odrebne_ustawy\", # 11\n",
" 'Dochody_razem', # 12\n", " \"Dochody_razem\", # 12\n",
" 'Dochody_z_majatku', # 13\n", " \"Dochody_z_majatku\", # 13\n",
" 'Dochody_z_najmu_i_dzierzawy', # 14\n", " \"Dochody_z_najmu_i_dzierzawy\", # 14\n",
" 'Dochody_z_uslug', # 15\n", " \"Dochody_z_uslug\", # 15\n",
" 'Dochody_dofinansowanie_inwestycyjne', # 16\n", " \"Dochody_dofinansowanie_inwestycyjne\", # 16\n",
" 'Dochody_dofinansowanie_razem', # 17\n", " \"Dochody_dofinansowanie_razem\", # 17\n",
" 'Udzialy_w_podatkach_dochodowych_od_osob_fizycznych', # 18\n", " \"Udzialy_w_podatkach_dochodowych_od_osob_fizycznych\", # 18\n",
" 'Udzialy_w_podatkach_dochodowych_od_osob_prywatnych', # 19\n", " \"Udzialy_w_podatkach_dochodowych_od_osob_prywatnych\", # 19\n",
" 'Udzialy_w_podatkach_dochodowych_razem', # 20\n", " \"Udzialy_w_podatkach_dochodowych_razem\", # 20\n",
" 'Wplywy_z_innych_lokalnych_oplat', # 21\n", " \"Wplywy_z_innych_lokalnych_oplat\", # 21\n",
" 'Wplywy_z_oplaty_eksploatacyjnej', # 22\n", " \"Wplywy_z_oplaty_eksploatacyjnej\", # 22\n",
" 'Wplywy_z_oplaty_skarbowej', # 23\n", " \"Wplywy_z_oplaty_skarbowej\", # 23\n",
" 'Wplywy_z_oplaty_targowej', # 24\n", " \"Wplywy_z_oplaty_targowej\", # 24\n",
" 'Ludnosc_ogolem', # 25\n", " \"Ludnosc_ogolem\", # 25\n",
" 'Ludnosc_w_wieku_poprodukcyjnym', # 26\n", " \"Ludnosc_w_wieku_poprodukcyjnym\", # 26\n",
" 'Ludnosc_w_wieku_produkcyjnym', # 27\n", " \"Ludnosc_w_wieku_produkcyjnym\", # 27\n",
" 'Ludnosc_w_wieku_produkcyjnym_mobilnym', # 28\n", " \"Ludnosc_w_wieku_produkcyjnym_mobilnym\", # 28\n",
" 'Ludnosc_w_wieku_produkcyjnym_niemobilnym', # 29\n", " \"Ludnosc_w_wieku_produkcyjnym_niemobilnym\", # 29\n",
" 'Ludnosc_w_wieku_przedprodukcyjnym', # 30\n", " \"Ludnosc_w_wieku_przedprodukcyjnym\", # 30\n",
" 'Ludnosc_mezczyzni', # 31\n", " \"Ludnosc_mezczyzni\", # 31\n",
" 'Ludnosc_mezczyzni_w_wieku_poprodukcyjnym', # 32\n", " \"Ludnosc_mezczyzni_w_wieku_poprodukcyjnym\", # 32\n",
" 'Ludnosc_mezczyzni_w_wieku_produkcyjnym', # 33\n", " \"Ludnosc_mezczyzni_w_wieku_produkcyjnym\", # 33\n",
" 'Ludnosc_mezczyzni_w_wieku_produkcyjnym_mobilnym', # 34\n", " \"Ludnosc_mezczyzni_w_wieku_produkcyjnym_mobilnym\", # 34\n",
" 'Ludnosc_mezczyzni_w_wieku_produkcyjnym_niemobilnym', # 35\n", " \"Ludnosc_mezczyzni_w_wieku_produkcyjnym_niemobilnym\", # 35\n",
" 'Ludnosc_mezczyzni_w_wieku_przedprodukcyjnym', # 36\n", " \"Ludnosc_mezczyzni_w_wieku_przedprodukcyjnym\", # 36\n",
" 'Ludnosc_kobiety', # 37\n", " \"Ludnosc_kobiety\", # 37\n",
" 'Ludnosc_kobiety_w_wieku_poprodukcyjnym', # 38\n", " \"Ludnosc_kobiety_w_wieku_poprodukcyjnym\", # 38\n",
" 'Ludnosc_kobiety_w_wieku_produkcyjnym', # 39\n", " \"Ludnosc_kobiety_w_wieku_produkcyjnym\", # 39\n",
" 'Ludnosc_kobiety_w_wieku_produkcyjnym_mobilnym', # 40\n", " \"Ludnosc_kobiety_w_wieku_produkcyjnym_mobilnym\", # 40\n",
" 'Ludnosc_kobiety_w_wieku_produkcyjnym_niemobilnym', # 41\n", " \"Ludnosc_kobiety_w_wieku_produkcyjnym_niemobilnym\", # 41\n",
" 'Ludnosc_kobiety_w_wieku_przedprodukcyjnym', # 42\n", " \"Ludnosc_kobiety_w_wieku_przedprodukcyjnym\", # 42\n",
" 'Wojewodztwo_Dolnoslaskie', # 43\n", " \"Wojewodztwo_Dolnoslaskie\", # 43\n",
" 'Wojewodztwo_Kujawsko_Pomorskie', # 44\n", " \"Wojewodztwo_Kujawsko_Pomorskie\", # 44\n",
" 'Wojewodztwo_Lubelskie', # 45\n", " \"Wojewodztwo_Lubelskie\", # 45\n",
" 'Wojewodztwo_Lubuskie', # 46\n", " \"Wojewodztwo_Lubuskie\", # 46\n",
" 'Wojewodztwo_Lodzkie', # 47\n", " \"Wojewodztwo_Lodzkie\", # 47\n",
" 'Wojewodztwo_Malopolskie', # 48\n", " \"Wojewodztwo_Malopolskie\", # 48\n",
" 'Wojewodztwo_Mazowieckie', # 49\n", " \"Wojewodztwo_Mazowieckie\", # 49\n",
" 'Wojewodztwo_Opolskie', # 50\n", " \"Wojewodztwo_Opolskie\", # 50\n",
" 'Wojewodztwo_Podkarpackie', # 51\n", " \"Wojewodztwo_Podkarpackie\", # 51\n",
" 'Wojewodztwo_Podlaskie', # 52\n", " \"Wojewodztwo_Podlaskie\", # 52\n",
" 'Wojewodztwo_Pomorskie', # 53\n", " \"Wojewodztwo_Pomorskie\", # 53\n",
" 'Wojewodztwo_Slaskie', # 54\n", " \"Wojewodztwo_Slaskie\", # 54\n",
" 'Wojewodztwo_Swietokrzyskie', # 55\n", " \"Wojewodztwo_Swietokrzyskie\", # 55\n",
" 'Wojewodztwo_Warminsko_Mazurskie', # 56\n", " \"Wojewodztwo_Warminsko_Mazurskie\", # 56\n",
" 'Wojewodztwo_Wielkopolskie', # 57\n", " \"Wojewodztwo_Wielkopolskie\", # 57\n",
" 'Wojewodztwo_Zachodniopomorskie', # 58\n", " \"Wojewodztwo_Zachodniopomorskie\", # 58\n",
" 'Gestosc_zaludnienia', # 59\n", " \"Gestosc_zaludnienia\", # 59\n",
" 'Ludnosc_na_1_km2', # 60\n", " \"Ludnosc_na_1_km2\", # 60\n",
" 'Ludnosc', # 61\n", " \"Ludnosc\", # 61\n",
" 'Wskaznik_urbanizacji', # 62\n", " \"Wskaznik_urbanizacji\", # 62\n",
" 'Zmiana_liczby_ludnosci', # 63\n", " \"Zmiana_liczby_ludnosci\", # 63\n",
" 'Saldo_migracji_na_1000_ludnosci', # 64\n", " \"Saldo_migracji_na_1000_ludnosci\", # 64\n",
" 'Saldo_migracji', # 65\n", " \"Saldo_migracji\", # 65\n",
" 'Wymeldowania_do_miast_kobiety', # 66\n", " \"Wymeldowania_do_miast_kobiety\", # 66\n",
" 'Wymeldowania_do_miast_mezczyzni', # 67\n", " \"Wymeldowania_do_miast_mezczyzni\", # 67\n",
" 'Wymeldowania_do_miast_ogolem', # 68\n", " \"Wymeldowania_do_miast_ogolem\", # 68\n",
" 'Wymeldowania_na_wies_kobiety', # 69\n", " \"Wymeldowania_na_wies_kobiety\", # 69\n",
" 'Wymeldowania_na_wies_mezczyzni', # 70\n", " \"Wymeldowania_na_wies_mezczyzni\", # 70\n",
" 'Wymeldowania_na_wies_ogolem', # 71\n", " \"Wymeldowania_na_wies_ogolem\", # 71\n",
" 'Wymeldowania_kobiety', # 72\n", " \"Wymeldowania_kobiety\", # 72\n",
" 'Wymeldowania_mezczyzni', # 73\n", " \"Wymeldowania_mezczyzni\", # 73\n",
" 'Wymeldowania_ogolem', # 74\n", " \"Wymeldowania_ogolem\", # 74\n",
" 'Zameldowania_kobiety', # 75\n", " \"Zameldowania_kobiety\", # 75\n",
" 'Zameldowania_mezczyzni', # 76\n", " \"Zameldowania_mezczyzni\", # 76\n",
" 'Zameldowania_ogolem', # 77\n", " \"Zameldowania_ogolem\", # 77\n",
" 'Zameldowania_z_miast_kobiety', # 78\n", " \"Zameldowania_z_miast_kobiety\", # 78\n",
" 'Zameldowania_z_miast_mezczyzni', # 79\n", " \"Zameldowania_z_miast_mezczyzni\", # 79\n",
" 'Zameldowania_z_miast_ogolem', # 80\n", " \"Zameldowania_z_miast_ogolem\", # 80\n",
" 'Zameldowania_ze_wsi_kobiety', # 81\n", " \"Zameldowania_ze_wsi_kobiety\", # 81\n",
" 'Zameldowania_ze_wsi_mezczyzni', # 82\n", " \"Zameldowania_ze_wsi_mezczyzni\", # 82\n",
" 'Zameldowania_ze_wsi_ogolem', # 83\n", " \"Zameldowania_ze_wsi_ogolem\", # 83\n",
" 'Miejsca_noclegowe_caloroczne', # 84\n", " \"Miejsca_noclegowe_caloroczne\", # 84\n",
" 'Miejsca_noclegowe_ogolem', # 85\n", " \"Miejsca_noclegowe_ogolem\", # 85\n",
" 'Obiekty_caloroczne', # 86\n", " \"Obiekty_caloroczne\", # 86\n",
" 'Obiekty_ogolem', # 87\n", " \"Obiekty_ogolem\", # 87\n",
" 'Turysci_ogolem', # 88\n", " \"Turysci_ogolem\", # 88\n",
" 'Turysci_zagraniczni', # 89\n", " \"Turysci_zagraniczni\", # 89\n",
" 'Bezrobotni_do_25_roku_zycia', # 90\n", " \"Bezrobotni_do_25_roku_zycia\", # 90\n",
" 'Dlugotrwale_bezrobotni', # 91\n", " \"Dlugotrwale_bezrobotni\", # 91\n",
" 'Bezrobotne_kobiety', # 92\n", " \"Bezrobotne_kobiety\", # 92\n",
" 'Bezrobotni_mezczyzni', # 93\n", " \"Bezrobotni_mezczyzni\", # 93\n",
" 'Bezrobotni_ogolem', # 94\n", " \"Bezrobotni_ogolem\", # 94\n",
" 'Bezrobotni_powyzej_50_roku_zycia', # 95\n", " \"Bezrobotni_powyzej_50_roku_zycia\", # 95\n",
" 'Gmina_miejska', # 96\n", " \"Gmina_miejska\", # 96\n",
" 'Gmina_miejsko_wiejska', # 97\n", " \"Gmina_miejsko_wiejska\", # 97\n",
" 'Gmina_wiejska', # 98\n", " \"Gmina_wiejska\", # 98\n",
" 'Odleglosc_Warszawa', # 99\n", " \"Odleglosc_Warszawa\", # 99\n",
" 'Odleglosc_od_centrum_decyzyjnego'] # 100\n", " \"Odleglosc_od_centrum_decyzyjnego\",\n",
"] # 100\n",
"\n", "\n",
"all_columns = ['Kod', 'Rok'] + feature_names\n", "all_columns = [\"Kod\", \"Rok\"] + feature_names\n",
"\n", "\n",
"df_data = pd.read_csv(\n", "df_data = pd.read_csv(\"dane1.csv\", encoding=\"ISO-8859-2\", converters={\"Kod\": str})\n",
" 'dane1.csv',\n",
" encoding='ISO-8859-2',\n",
" converters={'Kod': str})\n",
"\n", "\n",
"df_odl = pd.read_csv(\n", "df_odl = pd.read_csv(\n",
" 'gminy_centroid.csv',\n", " \"gminy_centroid.csv\", encoding=\"ISO-8859-2\", converters={\"TERYT\": str}\n",
" encoding='ISO-8859-2',\n", ")\n",
" converters={'TERYT': str})\n", "df_odl[\"TERYT\"] = df_odl[\"TERYT\"].astype(\"str\")\n",
"df_odl['TERYT'] = df_odl['TERYT'].astype('str')\n", "df_odl = df_odl[[\"TERYT\", \"odl_Wawa\", \"odl_woj\"]]\n",
"df_odl = df_odl[['TERYT', 'odl_Wawa', 'odl_woj']]\n", "df_odl = df_odl.rename(\n",
"df_odl = df_odl.rename(columns={\n", " columns={\n",
" 'TERYT': 'Kod',\n", " \"TERYT\": \"Kod\",\n",
" 'odl_Wawa': 'Odleglosc_Warszawa',\n", " \"odl_Wawa\": \"Odleglosc_Warszawa\",\n",
" 'odl_woj': 'Odleglosc_od_centrum_decyzyjnego'})\n", " \"odl_woj\": \"Odleglosc_od_centrum_decyzyjnego\",\n",
" }\n",
")\n",
"\n", "\n",
"df_data = df_data.merge(df_odl, on=['Kod'], how='left')\n", "df_data = df_data.merge(df_odl, on=[\"Kod\"], how=\"left\")\n",
"\n", "\n",
"print(df_data['Program_operacyjny'].drop_duplicates().values)" "print(df_data[\"Program_operacyjny\"].drop_duplicates().values)"
] ]
}, },
{ {
@ -181,37 +182,45 @@
], ],
"source": [ "source": [
"# Wybór programu operacyjnego...\n", "# Wybór programu operacyjnego...\n",
"df_data = df_data.loc[df_data['Program_operacyjny'] == 'Program Operacyjny Polska Cyfrowa'].reset_index(drop=True)\n", "df_data = df_data.loc[\n",
" df_data[\"Program_operacyjny\"] == \"Program Operacyjny Polska Cyfrowa\"\n",
"].reset_index(drop=True)\n",
"\n", "\n",
"# Uzupełnienie brakujących danych...\n", "# Uzupełnienie brakujących danych...\n",
"df_data[[\n", "df_data[\n",
" 'Miejsca_noclegowe_caloroczne',\n", " [\n",
" 'Miejsca_noclegowe_ogolem',\n", " \"Miejsca_noclegowe_caloroczne\",\n",
" 'Obiekty_caloroczne',\n", " \"Miejsca_noclegowe_ogolem\",\n",
" 'Obiekty_ogolem',\n", " \"Obiekty_caloroczne\",\n",
" 'Turysci_ogolem',\n", " \"Obiekty_ogolem\",\n",
" 'Turysci_zagraniczni']] = df_data[[\n", " \"Turysci_ogolem\",\n",
" 'Miejsca_noclegowe_caloroczne',\n", " \"Turysci_zagraniczni\",\n",
" 'Miejsca_noclegowe_ogolem',\n", " ]\n",
" 'Obiekty_caloroczne',\n", "] = df_data[\n",
" 'Obiekty_ogolem',\n", " [\n",
" 'Turysci_ogolem',\n", " \"Miejsca_noclegowe_caloroczne\",\n",
" 'Turysci_zagraniczni']].fillna(0)\n", " \"Miejsca_noclegowe_ogolem\",\n",
" \"Obiekty_caloroczne\",\n",
" \"Obiekty_ogolem\",\n",
" \"Turysci_ogolem\",\n",
" \"Turysci_zagraniczni\",\n",
" ]\n",
"].fillna(0)\n",
"\n", "\n",
"# Usunięcie niepotrzebnych rzędów...\n", "# Usunięcie niepotrzebnych rzędów...\n",
"df_data.dropna(inplace=True)\n", "df_data.dropna(inplace=True)\n",
"df_data = df_data[df_data['Suma'] > 0]\n", "df_data = df_data[df_data[\"Suma\"] > 0]\n",
"\n", "\n",
"df_data = df_data.groupby(all_columns)['Suma'].sum().reset_index()\n", "df_data = df_data.groupby(all_columns)[\"Suma\"].sum().reset_index()\n",
"\n", "\n",
"# ...\n", "# ...\n",
"# df_data['Suma'] = df_data['Suma'] / df_data['Ludnosc']\n", "# df_data['Suma'] = df_data['Suma'] / df_data['Ludnosc']\n",
"\n", "\n",
"print('Median', median(df_data['Suma']))\n", "print(\"Median\", median(df_data[\"Suma\"]))\n",
"print('Mean', sum(df_data['Suma'])/len(df_data['Suma']))\n", "print(\"Mean\", sum(df_data[\"Suma\"]) / len(df_data[\"Suma\"]))\n",
"\n", "\n",
"X = df_data[feature_names]\n", "X = df_data[feature_names]\n",
"y = df_data['Suma']\n", "y = df_data[\"Suma\"]\n",
"\n", "\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)" "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)"
] ]
@ -724,8 +733,55 @@
], ],
"source": [ "source": [
"param_grid = {\n", "param_grid = {\n",
" 'max_depth': [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28],\n", " \"max_depth\": [\n",
" 'n_features_to_select': [20, 19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2]\n", " 3,\n",
" 4,\n",
" 5,\n",
" 6,\n",
" 7,\n",
" 8,\n",
" 9,\n",
" 10,\n",
" 11,\n",
" 12,\n",
" 13,\n",
" 14,\n",
" 15,\n",
" 16,\n",
" 17,\n",
" 18,\n",
" 19,\n",
" 20,\n",
" 21,\n",
" 22,\n",
" 23,\n",
" 24,\n",
" 25,\n",
" 26,\n",
" 27,\n",
" 28,\n",
" ],\n",
" \"n_features_to_select\": [\n",
" 20,\n",
" 19,\n",
" 18,\n",
" 17,\n",
" 16,\n",
" 15,\n",
" 14,\n",
" 13,\n",
" 12,\n",
" 11,\n",
" 10,\n",
" 9,\n",
" 8,\n",
" 7,\n",
" 6,\n",
" 5,\n",
" 4,\n",
" 3,\n",
" 2,\n",
" ],\n",
"}\n", "}\n",
"\n", "\n",
"best_mse = np.inf\n", "best_mse = np.inf\n",
@ -735,41 +791,51 @@
"results_train = []\n", "results_train = []\n",
"results_test = []\n", "results_test = []\n",
"\n", "\n",
"for params in [{'max_depth': max_depth, 'n_features_to_select': n_features} \n", "for params in [\n",
" for max_depth in param_grid['max_depth'] \n", " {\"max_depth\": max_depth, \"n_features_to_select\": n_features}\n",
" for n_features in param_grid['n_features_to_select']]:\n", " for max_depth in param_grid[\"max_depth\"]\n",
"\n", " for n_features in param_grid[\"n_features_to_select\"]\n",
" selector = RFE(estimator=DecisionTreeRegressor(random_state=0, max_depth=params['max_depth']), n_features_to_select=params['n_features_to_select'])\n", "]:\n",
" selector = RFE(\n",
" estimator=DecisionTreeRegressor(random_state=0, max_depth=params[\"max_depth\"]),\n",
" n_features_to_select=params[\"n_features_to_select\"],\n",
" )\n",
" selector.fit(X_train, y_train)\n", " selector.fit(X_train, y_train)\n",
" X_train_selected = selector.transform(X_train)\n", " X_train_selected = selector.transform(X_train)\n",
" X_test_selected = selector.transform(X_test)\n", " X_test_selected = selector.transform(X_test)\n",
" \n", "\n",
" model = DecisionTreeRegressor(random_state=0, max_depth=params['max_depth'])\n", " model = DecisionTreeRegressor(random_state=0, max_depth=params[\"max_depth\"])\n",
" model.fit(X_train_selected, y_train)\n", " model.fit(X_train_selected, y_train)\n",
" \n", "\n",
" y_pred_train = model.predict(X_train_selected)\n", " y_pred_train = model.predict(X_train_selected)\n",
" y_pred_test = model.predict(X_test_selected)\n", " y_pred_test = model.predict(X_test_selected)\n",
"\n", "\n",
" mse_train = mean_squared_error(y_pred_train, y_train)\n", " mse_train = mean_squared_error(y_pred_train, y_train)\n",
" mse_test = mean_squared_error(y_pred_test, y_test)\n", " mse_test = mean_squared_error(y_pred_test, y_test)\n",
"\n", "\n",
" results_train.append((params['max_depth'], params['n_features_to_select'], mse_train))\n", " results_train.append(\n",
" results_test.append((params['max_depth'], params['n_features_to_select'], mse_test))\n", " (params[\"max_depth\"], params[\"n_features_to_select\"], mse_train)\n",
" )\n",
" results_test.append((params[\"max_depth\"], params[\"n_features_to_select\"], mse_test))\n",
"\n", "\n",
" if mse_test < best_mse:\n", " if mse_test < best_mse:\n",
" best_mse = mse_test\n", " best_mse = mse_test\n",
" best_model = (model, selector)\n", " best_model = (model, selector)\n",
" best_params = (params['max_depth'], params['n_features_to_select'])\n", " best_params = (params[\"max_depth\"], params[\"n_features_to_select\"])\n",
"\n", "\n",
" print(f\"max_depth: {params['max_depth']:>{2}}, \"\n", " print(\n",
" f\"n_features: {params['n_features_to_select']:>{2}}, \"\n", " f\"max_depth: {params['max_depth']:>{2}}, \"\n",
" f\"mse_train: {mse_train:>{20}.1f}, \"\n", " f\"n_features: {params['n_features_to_select']:>{2}}, \"\n",
" f\"mse_test: {mse_test:>{20}.1f} <-\")\n", " f\"mse_train: {mse_train:>{20}.1f}, \"\n",
" f\"mse_test: {mse_test:>{20}.1f} <-\"\n",
" )\n",
" else:\n", " else:\n",
" print(f\"max_depth: {params['max_depth']:>{2}}, \"\n", " print(\n",
" f\"n_features: {params['n_features_to_select']:>{2}}, \"\n", " f\"max_depth: {params['max_depth']:>{2}}, \"\n",
" f\"mse_train: {mse_train:>{20}.1f}, \"\n", " f\"n_features: {params['n_features_to_select']:>{2}}, \"\n",
" f\"mse_test: {mse_test:>{20}.1f}\")" " f\"mse_train: {mse_train:>{20}.1f}, \"\n",
" f\"mse_test: {mse_test:>{20}.1f}\"\n",
" )"
] ]
}, },
{ {
@ -806,20 +872,22 @@
} }
], ],
"source": [ "source": [
"results_train_df = pd.DataFrame(results_train, columns=['max_depth', 'n_features_to_select', 'mse_train'])\n", "results_train_df = pd.DataFrame(\n",
" results_train, columns=[\"max_depth\", \"n_features_to_select\", \"mse_train\"]\n",
")\n",
"\n", "\n",
"fig = plt.figure()\n", "fig = plt.figure()\n",
"ax = fig.add_subplot(111, projection='3d')\n", "ax = fig.add_subplot(111, projection=\"3d\")\n",
"\n", "\n",
"xs = results_train_df['max_depth']\n", "xs = results_train_df[\"max_depth\"]\n",
"ys = results_train_df['n_features_to_select']\n", "ys = results_train_df[\"n_features_to_select\"]\n",
"zs = results_train_df['mse_train']\n", "zs = results_train_df[\"mse_train\"]\n",
"\n", "\n",
"ax.scatter(xs, ys, zs, c=zs, marker='o')\n", "ax.scatter(xs, ys, zs, c=zs, marker=\"o\")\n",
"\n", "\n",
"ax.set_xlabel('Max Depth')\n", "ax.set_xlabel(\"Max Depth\")\n",
"ax.set_ylabel('Number of Features')\n", "ax.set_ylabel(\"Number of Features\")\n",
"ax.set_zlabel('MSE')\n", "ax.set_zlabel(\"MSE\")\n",
"\n", "\n",
"plt.show()" "plt.show()"
] ]
@ -841,20 +909,22 @@
} }
], ],
"source": [ "source": [
"results_test_df = pd.DataFrame(results_test, columns=['max_depth', 'n_features_to_select', 'mse_test'])\n", "results_test_df = pd.DataFrame(\n",
" results_test, columns=[\"max_depth\", \"n_features_to_select\", \"mse_test\"]\n",
")\n",
"\n", "\n",
"fig = plt.figure()\n", "fig = plt.figure()\n",
"ax = fig.add_subplot(111, projection='3d')\n", "ax = fig.add_subplot(111, projection=\"3d\")\n",
"\n", "\n",
"xs = results_test_df['max_depth']\n", "xs = results_test_df[\"max_depth\"]\n",
"ys = results_test_df['n_features_to_select']\n", "ys = results_test_df[\"n_features_to_select\"]\n",
"zs = results_test_df['mse_test']\n", "zs = results_test_df[\"mse_test\"]\n",
"\n", "\n",
"ax.scatter(xs, ys, zs, c=zs, marker='o')\n", "ax.scatter(xs, ys, zs, c=zs, marker=\"o\")\n",
"\n", "\n",
"ax.set_xlabel('Max Depth')\n", "ax.set_xlabel(\"Max Depth\")\n",
"ax.set_ylabel('Number of Features')\n", "ax.set_ylabel(\"Number of Features\")\n",
"ax.set_zlabel('MSE')\n", "ax.set_zlabel(\"MSE\")\n",
"\n", "\n",
"plt.show()" "plt.show()"
] ]
@ -879,21 +949,27 @@
"X_test_selected = best_model[1].transform(X_test)\n", "X_test_selected = best_model[1].transform(X_test)\n",
"y_pred_test = best_model[0].predict(X_test_selected)\n", "y_pred_test = best_model[0].predict(X_test_selected)\n",
"\n", "\n",
"color_column = df_data['Gmina_miejska'] * 0 + df_data['Gmina_miejsko_wiejska'] * 10 + df_data['Gmina_wiejska'] * 5\n", "color_column = (\n",
" df_data[\"Gmina_miejska\"] * 0\n",
" + df_data[\"Gmina_miejsko_wiejska\"] * 10\n",
" + df_data[\"Gmina_wiejska\"] * 5\n",
")\n",
"\n", "\n",
"color_column_train, color_column_test = train_test_split(color_column, test_size=0.2, random_state=0)\n", "color_column_train, color_column_test = train_test_split(\n",
" color_column, test_size=0.2, random_state=0\n",
")\n",
"\n", "\n",
"min_val = min(min(y_test), min(y_pred_test))\n", "min_val = min(min(y_test), min(y_pred_test))\n",
"max_val = max(max(y_test), max(y_pred_test))\n", "max_val = max(max(y_test), max(y_pred_test))\n",
"plt.plot([min_val, max_val], [min_val, max_val], 'r--')\n", "plt.plot([min_val, max_val], [min_val, max_val], \"r--\")\n",
"\n", "\n",
"plt.scatter(y_test, y_pred_test, alpha=0.5, c=color_column_test, cmap='rainbow')\n", "plt.scatter(y_test, y_pred_test, alpha=0.5, c=color_column_test, cmap=\"rainbow\")\n",
"plt.xlabel('Actual')\n", "plt.xlabel(\"Actual\")\n",
"plt.ylabel('Predicted')\n", "plt.ylabel(\"Predicted\")\n",
"plt.title('Actual vs Predicted')\n", "plt.title(\"Actual vs Predicted\")\n",
"\n", "\n",
"plt.xlim(0, 0.75*10**7)\n", "plt.xlim(0, 0.75 * 10**7)\n",
"plt.ylim(min(y_pred_test), 0.75*10**7)\n", "plt.ylim(min(y_pred_test), 0.75 * 10**7)\n",
"\n", "\n",
"plt.show()" "plt.show()"
] ]
@ -915,15 +991,19 @@
} }
], ],
"source": [ "source": [
"results_train_df = pd.DataFrame(results_train, columns=['max_depth', 'n_features_to_select', 'mse_train'])\n", "results_train_df = pd.DataFrame(\n",
" results_train, columns=[\"max_depth\", \"n_features_to_select\", \"mse_train\"]\n",
")\n",
"\n", "\n",
"heatmap_data = results_train_df.pivot(index='max_depth', columns='n_features_to_select', values='mse_train')\n", "heatmap_data = results_train_df.pivot(\n",
" index=\"max_depth\", columns=\"n_features_to_select\", values=\"mse_train\"\n",
")\n",
"\n", "\n",
"sns.heatmap(heatmap_data, annot=False, fmt=\".2f\", cmap=\"viridis\")\n", "sns.heatmap(heatmap_data, annot=False, fmt=\".2f\", cmap=\"viridis\")\n",
"\n", "\n",
"plt.title('MSE')\n", "plt.title(\"MSE\")\n",
"plt.xlabel('Number of Features')\n", "plt.xlabel(\"Number of Features\")\n",
"plt.ylabel('Max Depth')\n", "plt.ylabel(\"Max Depth\")\n",
"plt.show()" "plt.show()"
] ]
}, },
@ -944,15 +1024,19 @@
} }
], ],
"source": [ "source": [
"results_test_df = pd.DataFrame(results_test, columns=['max_depth', 'n_features_to_select', 'mse_test'])\n", "results_test_df = pd.DataFrame(\n",
" results_test, columns=[\"max_depth\", \"n_features_to_select\", \"mse_test\"]\n",
")\n",
"\n", "\n",
"heatmap_data = results_test_df.pivot(index='max_depth', columns='n_features_to_select', values='mse_test')\n", "heatmap_data = results_test_df.pivot(\n",
" index=\"max_depth\", columns=\"n_features_to_select\", values=\"mse_test\"\n",
")\n",
"\n", "\n",
"sns.heatmap(heatmap_data, annot=False, fmt=\".2f\", cmap=\"viridis\")\n", "sns.heatmap(heatmap_data, annot=False, fmt=\".2f\", cmap=\"viridis\")\n",
"\n", "\n",
"plt.title('MSE')\n", "plt.title(\"MSE\")\n",
"plt.xlabel('Number of Features')\n", "plt.xlabel(\"Number of Features\")\n",
"plt.ylabel('Max Depth')\n", "plt.ylabel(\"Max Depth\")\n",
"plt.show()" "plt.show()"
] ]
}, },
@ -983,8 +1067,10 @@
], ],
"source": [ "source": [
"feature_importance = dict(zip(feature_names, best_model[0].feature_importances_))\n", "feature_importance = dict(zip(feature_names, best_model[0].feature_importances_))\n",
"for feature, importance in sorted(feature_importance.items(), key=lambda x: x[1], reverse=True):\n", "for feature, importance in sorted(\n",
" print(f'{importance:.4f} \\u2014 {feature}')" " feature_importance.items(), key=lambda x: x[1], reverse=True\n",
"):\n",
" print(f\"{importance:.4f} \\u2014 {feature}\")"
] ]
}, },
{ {
@ -1478,10 +1564,10 @@
} }
], ],
"source": [ "source": [
"X_test['Przewidziana Suma'] = y_pred_test\n", "X_test[\"Przewidziana Suma\"] = y_pred_test\n",
"X_test['Suma'] = y_test\n", "X_test[\"Suma\"] = y_test\n",
"display(X_test[np.abs(y_test - y_pred_test) < 1])\n", "display(X_test[np.abs(y_test - y_pred_test) < 1])\n",
"display(X_test['Suma'][np.abs(y_test - y_pred_test) < 1].drop_duplicates().values)" "display(X_test[\"Suma\"][np.abs(y_test - y_pred_test) < 1].drop_duplicates().values)"
] ]
}, },
{ {
@ -1510,7 +1596,10 @@
} }
], ],
"source": [ "source": [
"selector = RFE(estimator=DecisionTreeRegressor(random_state=0, max_depth=best_params[0]), n_features_to_select=best_params[1])\n", "selector = RFE(\n",
" estimator=DecisionTreeRegressor(random_state=0, max_depth=best_params[0]),\n",
" n_features_to_select=best_params[1],\n",
")\n",
"selector.fit(X, y)\n", "selector.fit(X, y)\n",
"X_selected = selector.transform(X)\n", "X_selected = selector.transform(X)\n",
"\n", "\n",
@ -1518,8 +1607,10 @@
"model.fit(X_selected, y)\n", "model.fit(X_selected, y)\n",
"\n", "\n",
"feature_importance = dict(zip(feature_names, model.feature_importances_))\n", "feature_importance = dict(zip(feature_names, model.feature_importances_))\n",
"for feature, importance in sorted(feature_importance.items(), key=lambda x: x[1], reverse=True):\n", "for feature, importance in sorted(\n",
" print(f'{importance:.4f} \\u2014 {feature}')" " feature_importance.items(), key=lambda x: x[1], reverse=True\n",
"):\n",
" print(f\"{importance:.4f} \\u2014 {feature}\")"
] ]
}, },
{ {

312
Programming/PBAD/pbad.ipynb
View File

@ -6,14 +6,11 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"import pandas as pd\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.tree import DecisionTreeRegressor, plot_tree, export_text\n",
"from sklearn.metrics import mean_squared_error\n",
"import matplotlib.pyplot as plt\n", "import matplotlib.pyplot as plt\n",
"from sklearn.preprocessing import StandardScaler\n", "import pandas as pd\n",
"from sklearn.preprocessing import MinMaxScaler\n", "from sklearn.metrics import mean_squared_error\n",
"from sklearn.decomposition import PCA" "from sklearn.model_selection import train_test_split\n",
"from sklearn.tree import DecisionTreeRegressor, export_text"
] ]
}, },
{ {
@ -23,108 +20,109 @@
"outputs": [], "outputs": [],
"source": [ "source": [
"feature_names = [\n", "feature_names = [\n",
" 'Powierzchnia', # 1\n", " \"Powierzchnia\", # 1\n",
" 'Wynagrodzenie_ogolem', # 2\n", " \"Wynagrodzenie_ogolem\", # 2\n",
" 'Wynagrodzenie_w_relacji_do_sredniej', # 3\n", " \"Wynagrodzenie_w_relacji_do_sredniej\", # 3\n",
" 'Dochody_podatek_lesny', # 4\n", " \"Dochody_podatek_lesny\", # 4\n",
" 'Dochody_podatek_PCC', # 5\n", " \"Dochody_podatek_PCC\", # 5\n",
" 'Dochody_podatek_od_dzialalnosci_gospodarczej', # 6\n", " \"Dochody_podatek_od_dzialalnosci_gospodarczej\", # 6\n",
" 'Dochody_podatek_od_nieruchomosci', # 7\n", " \"Dochody_podatek_od_nieruchomosci\", # 7\n",
" 'Dochody_podatek_od_spadkow', # 8\n", " \"Dochody_podatek_od_spadkow\", # 8\n",
" 'Dochody_podatek_od_srodkow_transportowych', # 9\n", " \"Dochody_podatek_od_srodkow_transportowych\", # 9\n",
" 'Dochody_podatek_rolny', # 10\n", " \"Dochody_podatek_rolny\", # 10\n",
" 'Dochody_podatek_odrebne_ustawy', # 11\n", " \"Dochody_podatek_odrebne_ustawy\", # 11\n",
" 'Dochody_razem', # 12\n", " \"Dochody_razem\", # 12\n",
" 'Dochody_z_majatku', # 13\n", " \"Dochody_z_majatku\", # 13\n",
" 'Dochody_z_najmu_i_dzierzawy', # 14\n", " \"Dochody_z_najmu_i_dzierzawy\", # 14\n",
" 'Dochody_z_uslug', # 15\n", " \"Dochody_z_uslug\", # 15\n",
" 'Dochody_dofinansowanie_inwestycyjne', # 16\n", " \"Dochody_dofinansowanie_inwestycyjne\", # 16\n",
" 'Dochody_dofinansowanie_razem', # 17\n", " \"Dochody_dofinansowanie_razem\", # 17\n",
" 'Udzialy_w_podatkach_dochodowych_od_osob_fizycznych', # 18\n", " \"Udzialy_w_podatkach_dochodowych_od_osob_fizycznych\", # 18\n",
" 'Udzialy_w_podatkach_dochodowych_od_osob_prywatnych', # 19\n", " \"Udzialy_w_podatkach_dochodowych_od_osob_prywatnych\", # 19\n",
" 'Udzialy_w_podatkach_dochodowych_razem', # 20\n", " \"Udzialy_w_podatkach_dochodowych_razem\", # 20\n",
" 'Wplywy_z_innych_lokalnych_oplat', # 21\n", " \"Wplywy_z_innych_lokalnych_oplat\", # 21\n",
" 'Wplywy_z_oplaty_eksploatacyjnej', # 22\n", " \"Wplywy_z_oplaty_eksploatacyjnej\", # 22\n",
" 'Wplywy_z_oplaty_skarbowej', # 23\n", " \"Wplywy_z_oplaty_skarbowej\", # 23\n",
" 'Wplywy_z_oplaty_targowej', # 24\n", " \"Wplywy_z_oplaty_targowej\", # 24\n",
" 'Ludnosc_ogolem', # 25\n", " \"Ludnosc_ogolem\", # 25\n",
" 'Ludnosc_w_wieku_poprodukcyjnym', # 26\n", " \"Ludnosc_w_wieku_poprodukcyjnym\", # 26\n",
" 'Ludnosc_w_wieku_produkcyjnym', # 27\n", " \"Ludnosc_w_wieku_produkcyjnym\", # 27\n",
" 'Ludnosc_w_wieku_produkcyjnym_mobilnym', # 28\n", " \"Ludnosc_w_wieku_produkcyjnym_mobilnym\", # 28\n",
" 'Ludnosc_w_wieku_produkcyjnym_niemobilnym', # 29\n", " \"Ludnosc_w_wieku_produkcyjnym_niemobilnym\", # 29\n",
" 'Ludnosc_w_wieku_przedprodukcyjnym', # 30\n", " \"Ludnosc_w_wieku_przedprodukcyjnym\", # 30\n",
" 'Ludnosc_mezczyzni', # 31\n", " \"Ludnosc_mezczyzni\", # 31\n",
" 'Ludnosc_mezczyzni_w_wieku_poprodukcyjnym', # 32\n", " \"Ludnosc_mezczyzni_w_wieku_poprodukcyjnym\", # 32\n",
" 'Ludnosc_mezczyzni_w_wieku_produkcyjnym', # 33\n", " \"Ludnosc_mezczyzni_w_wieku_produkcyjnym\", # 33\n",
" 'Ludnosc_mezczyzni_w_wieku_produkcyjnym_mobilnym', # 34\n", " \"Ludnosc_mezczyzni_w_wieku_produkcyjnym_mobilnym\", # 34\n",
" 'Ludnosc_mezczyzni_w_wieku_produkcyjnym_niemobilnym', # 35\n", " \"Ludnosc_mezczyzni_w_wieku_produkcyjnym_niemobilnym\", # 35\n",
" 'Ludnosc_mezczyzni_w_wieku_przedprodukcyjnym', # 36\n", " \"Ludnosc_mezczyzni_w_wieku_przedprodukcyjnym\", # 36\n",
" 'Ludnosc_kobiety', # 37\n", " \"Ludnosc_kobiety\", # 37\n",
" 'Ludnosc_kobiety_w_wieku_poprodukcyjnym', # 38\n", " \"Ludnosc_kobiety_w_wieku_poprodukcyjnym\", # 38\n",
" 'Ludnosc_kobiety_w_wieku_produkcyjnym', # 39\n", " \"Ludnosc_kobiety_w_wieku_produkcyjnym\", # 39\n",
" 'Ludnosc_kobiety_w_wieku_produkcyjnym_mobilnym', # 40\n", " \"Ludnosc_kobiety_w_wieku_produkcyjnym_mobilnym\", # 40\n",
" 'Ludnosc_kobiety_w_wieku_produkcyjnym_niemobilnym', # 41\n", " \"Ludnosc_kobiety_w_wieku_produkcyjnym_niemobilnym\", # 41\n",
" 'Ludnosc_kobiety_w_wieku_przedprodukcyjnym', # 42\n", " \"Ludnosc_kobiety_w_wieku_przedprodukcyjnym\", # 42\n",
" 'Wojewodztwo_Dolnoslaskie', # 43\n", " \"Wojewodztwo_Dolnoslaskie\", # 43\n",
" 'Wojewodztwo_Kujawsko_Pomorskie', # 44\n", " \"Wojewodztwo_Kujawsko_Pomorskie\", # 44\n",
" 'Wojewodztwo_Lubelskie', # 45\n", " \"Wojewodztwo_Lubelskie\", # 45\n",
" 'Wojewodztwo_Lubuskie', # 46\n", " \"Wojewodztwo_Lubuskie\", # 46\n",
" 'Wojewodztwo_Lodzkie', # 47\n", " \"Wojewodztwo_Lodzkie\", # 47\n",
" 'Wojewodztwo_Malopolskie', # 48\n", " \"Wojewodztwo_Malopolskie\", # 48\n",
" 'Wojewodztwo_Mazowieckie', # 49\n", " \"Wojewodztwo_Mazowieckie\", # 49\n",
" 'Wojewodztwo_Opolskie', # 50\n", " \"Wojewodztwo_Opolskie\", # 50\n",
" 'Wojewodztwo_Podkarpackie', # 51\n", " \"Wojewodztwo_Podkarpackie\", # 51\n",
" 'Wojewodztwo_Podlaskie', # 52\n", " \"Wojewodztwo_Podlaskie\", # 52\n",
" 'Wojewodztwo_Pomorskie', # 53\n", " \"Wojewodztwo_Pomorskie\", # 53\n",
" 'Wojewodztwo_Slaskie', # 54\n", " \"Wojewodztwo_Slaskie\", # 54\n",
" 'Wojewodztwo_Swietokrzyskie', # 55\n", " \"Wojewodztwo_Swietokrzyskie\", # 55\n",
" 'Wojewodztwo_Warminsko_Mazurskie', # 56\n", " \"Wojewodztwo_Warminsko_Mazurskie\", # 56\n",
" 'Wojewodztwo_Wielkopolskie', # 57\n", " \"Wojewodztwo_Wielkopolskie\", # 57\n",
" 'Wojewodztwo_Zachodniopomorskie', # 58\n", " \"Wojewodztwo_Zachodniopomorskie\", # 58\n",
" 'Gestosc_zaludnienia', # 59\n", " \"Gestosc_zaludnienia\", # 59\n",
" 'Ludnosc_na_1_km2', # 60\n", " \"Ludnosc_na_1_km2\", # 60\n",
" 'Ludnosc', # 61\n", " \"Ludnosc\", # 61\n",
" 'Wskaznik_urbanizacji', # 62\n", " \"Wskaznik_urbanizacji\", # 62\n",
" 'Zmiana_liczby_ludnosci', # 63\n", " \"Zmiana_liczby_ludnosci\", # 63\n",
" 'Saldo_migracji_na_1000_ludnosci', # 64\n", " \"Saldo_migracji_na_1000_ludnosci\", # 64\n",
" 'Saldo_migracji', # 65\n", " \"Saldo_migracji\", # 65\n",
" 'Wymeldowania_do_miast_kobiety', # 66\n", " \"Wymeldowania_do_miast_kobiety\", # 66\n",
" 'Wymeldowania_do_miast_mezczyzni', # 67\n", " \"Wymeldowania_do_miast_mezczyzni\", # 67\n",
" 'Wymeldowania_do_miast_ogolem', # 68\n", " \"Wymeldowania_do_miast_ogolem\", # 68\n",
" 'Wymeldowania_na_wies_kobiety', # 69\n", " \"Wymeldowania_na_wies_kobiety\", # 69\n",
" 'Wymeldowania_na_wies_mezczyzni', # 70\n", " \"Wymeldowania_na_wies_mezczyzni\", # 70\n",
" 'Wymeldowania_na_wies_ogolem', # 71\n", " \"Wymeldowania_na_wies_ogolem\", # 71\n",
" 'Wymeldowania_kobiety', # 72\n", " \"Wymeldowania_kobiety\", # 72\n",
" 'Wymeldowania_mezczyzni', # 73\n", " \"Wymeldowania_mezczyzni\", # 73\n",
" 'Wymeldowania_ogolem', # 74\n", " \"Wymeldowania_ogolem\", # 74\n",
" 'Zameldowania_kobiety', # 75\n", " \"Zameldowania_kobiety\", # 75\n",
" 'Zameldowania_mezczyzni', # 76\n", " \"Zameldowania_mezczyzni\", # 76\n",
" 'Zameldowania_ogolem', # 77\n", " \"Zameldowania_ogolem\", # 77\n",
" 'Zameldowania_z_miast_kobiety', # 78\n", " \"Zameldowania_z_miast_kobiety\", # 78\n",
" 'Zameldowania_z_miast_mezczyzni', # 79\n", " \"Zameldowania_z_miast_mezczyzni\", # 79\n",
" 'Zameldowania_z_miast_ogolem', # 80\n", " \"Zameldowania_z_miast_ogolem\", # 80\n",
" 'Zameldowania_ze_wsi_kobiety', # 81\n", " \"Zameldowania_ze_wsi_kobiety\", # 81\n",
" 'Zameldowania_ze_wsi_mezczyzni', # 82\n", " \"Zameldowania_ze_wsi_mezczyzni\", # 82\n",
" 'Zameldowania_ze_wsi_ogolem', # 83\n", " \"Zameldowania_ze_wsi_ogolem\", # 83\n",
" 'Miejsca_noclegowe_caloroczne', # 84\n", " \"Miejsca_noclegowe_caloroczne\", # 84\n",
" 'Miejsca_noclegowe_ogolem', # 85\n", " \"Miejsca_noclegowe_ogolem\", # 85\n",
" 'Obiekty_caloroczne', # 86\n", " \"Obiekty_caloroczne\", # 86\n",
" 'Obiekty_ogolem', # 87\n", " \"Obiekty_ogolem\", # 87\n",
" 'Turysci_ogolem', # 88\n", " \"Turysci_ogolem\", # 88\n",
" 'Turysci_zagraniczni', # 89\n", " \"Turysci_zagraniczni\", # 89\n",
" 'Bezrobotni_do_25_roku_zycia', # 90\n", " \"Bezrobotni_do_25_roku_zycia\", # 90\n",
" 'Dlugotrwale_bezrobotni', # 91\n", " \"Dlugotrwale_bezrobotni\", # 91\n",
" 'Bezrobotne_kobiety', # 92\n", " \"Bezrobotne_kobiety\", # 92\n",
" 'Bezrobotni_mezczyzni', # 93\n", " \"Bezrobotni_mezczyzni\", # 93\n",
" 'Bezrobotni_ogolem', # 94\n", " \"Bezrobotni_ogolem\", # 94\n",
" 'Bezrobotni_powyzej_50_roku_zycia', # 95\n", " \"Bezrobotni_powyzej_50_roku_zycia\", # 95\n",
" 'Gmina_miejska', # 96\n", " \"Gmina_miejska\", # 96\n",
" 'Gmina_miejsko_wiejska', # 97\n", " \"Gmina_miejsko_wiejska\", # 97\n",
" 'Gmina_wiejska', # 98\n", " \"Gmina_wiejska\", # 98\n",
" 'Odleglosc_Warszawa', # 99\n", " \"Odleglosc_Warszawa\", # 99\n",
" 'Odleglosc_od_centrum_decyzyjnego'] # 100\n", " \"Odleglosc_od_centrum_decyzyjnego\",\n",
"] # 100\n",
"\n", "\n",
"all_columns = ['Kod', 'Rok'] + feature_names" "all_columns = [\"Kod\", \"Rok\"] + feature_names"
] ]
}, },
{ {
@ -133,10 +131,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"df_data = pd.read_csv(\n", "df_data = pd.read_csv(\"dane1.csv\", encoding=\"ISO-8859-2\", converters={\"Kod\": str})"
" 'dane1.csv',\n",
" encoding='ISO-8859-2',\n",
" converters={'Kod': str})"
] ]
}, },
{ {
@ -146,15 +141,17 @@
"outputs": [], "outputs": [],
"source": [ "source": [
"df_odl = pd.read_csv(\n", "df_odl = pd.read_csv(\n",
" 'gminy_centroid.csv',\n", " \"gminy_centroid.csv\", encoding=\"ISO-8859-2\", converters={\"TERYT\": str}\n",
" encoding='ISO-8859-2',\n", ")\n",
" converters={'TERYT': str})\n", "df_odl[\"TERYT\"] = df_odl[\"TERYT\"].astype(\"str\")\n",
"df_odl['TERYT'] = df_odl['TERYT'].astype('str')\n", "df_odl = df_odl[[\"TERYT\", \"odl_Wawa\", \"odl_woj\"]]\n",
"df_odl = df_odl[['TERYT', 'odl_Wawa', 'odl_woj']]\n", "df_odl = df_odl.rename(\n",
"df_odl = df_odl.rename(columns={\n", " columns={\n",
" 'TERYT': 'Kod',\n", " \"TERYT\": \"Kod\",\n",
" 'odl_Wawa': 'Odleglosc_Warszawa',\n", " \"odl_Wawa\": \"Odleglosc_Warszawa\",\n",
" 'odl_woj': 'Odleglosc_od_centrum_decyzyjnego'})" " \"odl_woj\": \"Odleglosc_od_centrum_decyzyjnego\",\n",
" }\n",
")"
] ]
}, },
{ {
@ -163,7 +160,7 @@
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"df_data = df_data.merge(df_odl, on=['Kod'], how='left')" "df_data = df_data.merge(df_odl, on=[\"Kod\"], how=\"left\")"
] ]
}, },
{ {
@ -184,7 +181,7 @@
} }
], ],
"source": [ "source": [
"print(df_data['Program_operacyjny'].drop_duplicates().values)" "print(df_data[\"Program_operacyjny\"].drop_duplicates().values)"
] ]
}, },
{ {
@ -207,35 +204,44 @@
], ],
"source": [ "source": [
"# Wybór programu operacyjnego...\n", "# Wybór programu operacyjnego...\n",
"df_data = df_data.loc[df_data['Program_operacyjny'] == 'Program Operacyjny Infrastruktura i ?rodowisko 2014-2020'].reset_index(drop=True)\n", "df_data = df_data.loc[\n",
" df_data[\"Program_operacyjny\"]\n",
" == \"Program Operacyjny Infrastruktura i ?rodowisko 2014-2020\"\n",
"].reset_index(drop=True)\n",
"# df_data = df_data.loc[df_data['Program_operacyjny'] == 'Program Operacyjny Inteligentny RozwĂłj'].reset_index(drop=True)\n", "# df_data = df_data.loc[df_data['Program_operacyjny'] == 'Program Operacyjny Inteligentny RozwĂłj'].reset_index(drop=True)\n",
"# df_data = df_data.loc[df_data['Program_operacyjny'] == 'Program Operacyjny Polska Cyfrowa'].reset_index(drop=True)\n", "# df_data = df_data.loc[df_data['Program_operacyjny'] == 'Program Operacyjny Polska Cyfrowa'].reset_index(drop=True)\n",
"# df_data = df_data.loc[df_data['Program_operacyjny'] == 'Program Operacyjny Wiedza Edukacja RozwĂłj'].reset_index(drop=True)\n", "# df_data = df_data.loc[df_data['Program_operacyjny'] == 'Program Operacyjny Wiedza Edukacja RozwĂłj'].reset_index(drop=True)\n",
"# df_data = df_data.loc[df_data['Program_operacyjny'] == 'Program Operacyjny Polska Wschodnia'].reset_index(drop=True)\n", "# df_data = df_data.loc[df_data['Program_operacyjny'] == 'Program Operacyjny Polska Wschodnia'].reset_index(drop=True)\n",
"\n", "\n",
"# Uzupełnienie brakujących danych...\n", "# Uzupełnienie brakujących danych...\n",
"df_data[[\n", "df_data[\n",
" 'Miejsca_noclegowe_caloroczne',\n", " [\n",
" 'Miejsca_noclegowe_ogolem',\n", " \"Miejsca_noclegowe_caloroczne\",\n",
" 'Obiekty_caloroczne',\n", " \"Miejsca_noclegowe_ogolem\",\n",
" 'Obiekty_ogolem',\n", " \"Obiekty_caloroczne\",\n",
" 'Turysci_ogolem',\n", " \"Obiekty_ogolem\",\n",
" 'Turysci_zagraniczni']] = df_data[[\n", " \"Turysci_ogolem\",\n",
" 'Miejsca_noclegowe_caloroczne',\n", " \"Turysci_zagraniczni\",\n",
" 'Miejsca_noclegowe_ogolem',\n", " ]\n",
" 'Obiekty_caloroczne',\n", "] = df_data[\n",
" 'Obiekty_ogolem',\n", " [\n",
" 'Turysci_ogolem',\n", " \"Miejsca_noclegowe_caloroczne\",\n",
" 'Turysci_zagraniczni']].fillna(0)\n", " \"Miejsca_noclegowe_ogolem\",\n",
" \"Obiekty_caloroczne\",\n",
" \"Obiekty_ogolem\",\n",
" \"Turysci_ogolem\",\n",
" \"Turysci_zagraniczni\",\n",
" ]\n",
"].fillna(0)\n",
"\n", "\n",
"# Usunięcie niepotrzebnych rzędów...\n", "# Usunięcie niepotrzebnych rzędów...\n",
"df_data.dropna(inplace=True)\n", "df_data.dropna(inplace=True)\n",
"df_data = df_data[df_data['Suma'] > 0]\n", "df_data = df_data[df_data[\"Suma\"] > 0]\n",
"\n", "\n",
"df_data = df_data.groupby(all_columns)['Suma'].sum().reset_index()\n", "df_data = df_data.groupby(all_columns)[\"Suma\"].sum().reset_index()\n",
"\n", "\n",
"# ...\n", "# ...\n",
"df_data['Suma'] = df_data['Suma'] / df_data['Ludnosc']" "df_data[\"Suma\"] = df_data[\"Suma\"] / df_data[\"Ludnosc\"]"
] ]
}, },
{ {
@ -257,18 +263,20 @@
"# scaler = MinMaxScaler()\n", "# scaler = MinMaxScaler()\n",
"# X = scaler.fit_transform(X)\n", "# X = scaler.fit_transform(X)\n",
"\n", "\n",
"y = df_data['Suma']\n", "y = df_data[\"Suma\"]\n",
"\n", "\n",
"color_column = (df_data['Gestosc_zaludnienia'] > 1.5).astype(int)\n", "color_column = (df_data[\"Gestosc_zaludnienia\"] > 1.5).astype(int)\n",
"\n", "\n",
"X_train, X_test, y_train, y_test, color_column_train, color_column_test = train_test_split(X, y, color_column, test_size=0.2, random_state=99)\n", "X_train, X_test, y_train, y_test, color_column_train, color_column_test = (\n",
" train_test_split(X, y, color_column, test_size=0.2, random_state=99)\n",
")\n",
"\n", "\n",
"model = DecisionTreeRegressor(random_state=99)\n", "model = DecisionTreeRegressor(random_state=99)\n",
"model.fit(X_train, y_train)\n", "model.fit(X_train, y_train)\n",
"\n", "\n",
"y_pred = model.predict(X_test)\n", "y_pred = model.predict(X_test)\n",
"mse = mean_squared_error(y_test, y_pred)\n", "mse = mean_squared_error(y_test, y_pred)\n",
"print('Mean Squared Error:', mse)" "print(\"Mean Squared Error:\", mse)"
] ]
}, },
{ {
@ -288,10 +296,10 @@
} }
], ],
"source": [ "source": [
"plt.scatter(y_test, y_pred, alpha=0.5, c=color_column_test, cmap='viridis')\n", "plt.scatter(y_test, y_pred, alpha=0.5, c=color_column_test, cmap=\"viridis\")\n",
"plt.xlabel('Actual')\n", "plt.xlabel(\"Actual\")\n",
"plt.ylabel('Predicted')\n", "plt.ylabel(\"Predicted\")\n",
"plt.title('Actual vs Predicted')\n", "plt.title(\"Actual vs Predicted\")\n",
"\n", "\n",
"# plt.xlim(0, max(max(y_test), max(y_pred)))\n", "# plt.xlim(0, max(max(y_test), max(y_pred)))\n",
"# plt.ylim(0, max(max(y_test), max(y_pred)))\n", "# plt.ylim(0, max(max(y_test), max(y_pred)))\n",
@ -351,7 +359,7 @@
], ],
"source": [ "source": [
"correlation_matrix = df_data.corr()\n", "correlation_matrix = df_data.corr()\n",
"print(correlation_matrix['Suma'].sort_values(ascending=False)[1:29])" "print(correlation_matrix[\"Suma\"].sort_values(ascending=False)[1:29])"
] ]
}, },
{ {
@ -792,8 +800,10 @@
], ],
"source": [ "source": [
"feature_importance = dict(zip(feature_names, model.feature_importances_))\n", "feature_importance = dict(zip(feature_names, model.feature_importances_))\n",
"for feature, importance in sorted(feature_importance.items(), key=lambda x: x[1], reverse=True):\n", "for feature, importance in sorted(\n",
" print(f'{importance:.4f} \\u2014 {feature}')" " feature_importance.items(), key=lambda x: x[1], reverse=True\n",
"):\n",
" print(f\"{importance:.4f} \\u2014 {feature}\")"
] ]
} }
], ],

11
Programming/PSD/flink/anomaly_detector.py
View File

@ -1,18 +1,21 @@
import json
from pyflink.common.serialization import SimpleStringSchema
from pyflink.datastream import StreamExecutionEnvironment from pyflink.datastream import StreamExecutionEnvironment
from pyflink.datastream.connectors.kafka import FlinkKafkaConsumer from pyflink.datastream.connectors.kafka import FlinkKafkaConsumer
from pyflink.common.serialization import SimpleStringSchema
import json
def detect_anomalies(transaction): def detect_anomalies(transaction):
transaction_data = json.loads(transaction) transaction_data = json.loads(transaction)
# Add anomaly detection logic here # Add anomaly detection logic here
return transaction_data return transaction_data
env = StreamExecutionEnvironment.get_execution_environment() env = StreamExecutionEnvironment.get_execution_environment()
kafka_consumer = FlinkKafkaConsumer( kafka_consumer = FlinkKafkaConsumer(
topics='transactions', topics="transactions",
deserialization_schema=SimpleStringSchema(), deserialization_schema=SimpleStringSchema(),
properties={'bootstrap.servers': 'localhost:9092', 'group.id': 'anomaly_detection'} properties={"bootstrap.servers": "localhost:9092", "group.id": "anomaly_detection"},
) )
data_stream = env.add_source(kafka_consumer).map(detect_anomalies) data_stream = env.add_source(kafka_consumer).map(detect_anomalies)

23
Programming/PSD/kafka/generate_and_send.py
View File

@ -1,32 +1,41 @@
from confluent_kafka import Producer
import json import json
import random import random
import time import time
from confluent_kafka import Producer
# Configuration for Kafka Producer # Configuration for Kafka Producer
conf = {'bootstrap.servers': "localhost:9092"} conf = {"bootstrap.servers": "localhost:9092"}
producer = Producer(**conf) producer = Producer(**conf)
def generate_transaction(): def generate_transaction():
return { return {
"card_id": random.randint(1000, 9999), "card_id": random.randint(1000, 9999),
"user_id": random.randint(100, 999), "user_id": random.randint(100, 999),
"location": { "location": {
"latitude": round(random.uniform(-90, 90), 6), "latitude": round(random.uniform(-90, 90), 6),
"longitude": round(random.uniform(-180, 180), 6) "longitude": round(random.uniform(-180, 180), 6),
}, },
"transaction_value": round(random.uniform(1, 1000), 2), "transaction_value": round(random.uniform(1, 1000), 2),
"spending_limit": round(random.uniform(1000, 5000), 2) "spending_limit": round(random.uniform(1000, 5000), 2),
} }
def delivery_report(err, msg): def delivery_report(err, msg):
if err is not None: if err is not None:
print('Message delivery failed: {}'.format(err)) print(f"Message delivery failed: {err}")
else: else:
print('Message delivered to {} [{}]'.format(msg.topic(), msg.partition())) print(f"Message delivered to {msg.topic()} [{msg.partition()}]")
while True: while True:
transaction = generate_transaction() transaction = generate_transaction()
producer.produce('transactions', key=str(transaction["card_id"]), value=json.dumps(transaction), callback=delivery_report) producer.produce(
"transactions",
key=str(transaction["card_id"]),
value=json.dumps(transaction),
callback=delivery_report,
)
producer.poll(1) producer.poll(1)
time.sleep(1) # Adjust the sleep time to control the frequency of transactions time.sleep(1) # Adjust the sleep time to control the frequency of transactions

12
Programming/PSD/kafka/kafka_consumer.py
View File

@ -1,11 +1,13 @@
from confluent_kafka import Consumer, KafkaError from confluent_kafka import Consumer, KafkaError
conf = {'bootstrap.servers': "localhost:9092", conf = {
'group.id': "test_group", "bootstrap.servers": "localhost:9092",
'auto.offset.reset': 'earliest'} "group.id": "test_group",
"auto.offset.reset": "earliest",
}
consumer = Consumer(**conf) consumer = Consumer(**conf)
consumer.subscribe(['transactions']) consumer.subscribe(["transactions"])
while True: while True:
msg = consumer.poll(timeout=1.0) msg = consumer.poll(timeout=1.0)
@ -17,5 +19,5 @@ while True:
else: else:
print(msg.error()) print(msg.error())
break break
print('Received message: {}'.format(msg.value().decode('utf-8'))) print("Received message: {}".format(msg.value().decode("utf-8")))
consumer.close() consumer.close()

23
Programming/PSD/webinterface/webinterface.py
View File

@ -1,17 +1,20 @@
from flask import Flask, jsonify
from confluent_kafka import Consumer, KafkaError from confluent_kafka import Consumer, KafkaError
from flask import Flask, jsonify
app = Flask(__name__) app = Flask(__name__)
@app.route('/alarms', methods=['GET'])
@app.route("/alarms", methods=["GET"])
def get_alarms(): def get_alarms():
alarms = [] alarms = []
conf = {'bootstrap.servers': "localhost:9092", conf = {
'group.id': "alarm_group", "bootstrap.servers": "localhost:9092",
'auto.offset.reset': 'earliest'} "group.id": "alarm_group",
"auto.offset.reset": "earliest",
}
consumer = Consumer(**conf) consumer = Consumer(**conf)
consumer.subscribe(['alarms']) consumer.subscribe(["alarms"])
while True: while True:
msg = consumer.poll(timeout=1.0) msg = consumer.poll(timeout=1.0)
@ -20,12 +23,12 @@ def get_alarms():
if msg.error(): if msg.error():
if msg.error().code() == KafkaError._PARTITION_EOF: if msg.error().code() == KafkaError._PARTITION_EOF:
continue continue
else: return str(msg.error()), 500
return str(msg.error()), 500 alarms.append(json.loads(msg.value().decode("utf-8")))
alarms.append(json.loads(msg.value().decode('utf-8')))
consumer.close() consumer.close()
return jsonify(alarms) return jsonify(alarms)
if __name__ == '__main__':
if __name__ == "__main__":
app.run(debug=True) app.run(debug=True)

13
Programming/PSD/zin1/konsument.py
View File

@ -1,12 +1,14 @@
import json import json
from kafka import KafkaConsumer from kafka import KafkaConsumer
# Thresholds # Thresholds
TEMP_TOO_COLD = -10 TEMP_TOO_COLD = -10
TEMP_TOO_HOT = 35 TEMP_TOO_HOT = 35
def process_temperature_reading(reading): def process_temperature_reading(reading):
temperature = reading['temperature'] temperature = reading["temperature"]
if temperature < TEMP_TOO_COLD: if temperature < TEMP_TOO_COLD:
alert = f"WARNING: Temperature is too cold! ({temperature}°C)" alert = f"WARNING: Temperature is too cold! ({temperature}°C)"
elif temperature > TEMP_TOO_HOT: elif temperature > TEMP_TOO_HOT:
@ -15,11 +17,12 @@ def process_temperature_reading(reading):
alert = f"Temperature is normal. ({temperature}°C)" alert = f"Temperature is normal. ({temperature}°C)"
return alert return alert
if __name__ == '__main__':
if __name__ == "__main__":
consumer = KafkaConsumer( consumer = KafkaConsumer(
'temperature_readings', "temperature_readings",
bootstrap_servers='localhost:9092', bootstrap_servers="localhost:9092",
auto_offset_reset='earliest' auto_offset_reset="earliest",
) )
for message in consumer: for message in consumer:

12
Programming/PSD/zin1/producent.py
View File

@ -1,20 +1,22 @@
import json import json
import random import random
import time import time
from kafka import KafkaProducer from kafka import KafkaProducer
from simulate_temperature_sensor import generate_temperature_reading from simulate_temperature_sensor import generate_temperature_reading
def serializer(message): def serializer(message):
return json.dumps(message).encode('utf-8') return json.dumps(message).encode("utf-8")
producer = KafkaProducer( producer = KafkaProducer(
bootstrap_servers=['localhost:9092'], bootstrap_servers=["localhost:9092"], value_serializer=serializer
value_serializer=serializer
) )
if __name__ == '__main__': if __name__ == "__main__":
while True: while True:
reading = generate_temperature_reading() reading = generate_temperature_reading()
print(f"Sending reading: {reading}") print(f"Sending reading: {reading}")
producer.send('temperature_readings', reading) producer.send("temperature_readings", reading)
time.sleep(random.randint(1, 5)) # Simulate readings sent at random intervals time.sleep(random.randint(1, 5)) # Simulate readings sent at random intervals

11
Programming/PSD/zin1/simulate_temperature_sensor.py
View File

@ -1,12 +1,15 @@
import random import random
import time import time
def generate_temperature_reading(): def generate_temperature_reading():
location_id = random.randint(1, 10) # Simulate 10 different locations location_id = random.randint(1, 10) # Simulate 10 different locations
temperature = random.uniform(-20, 40) # Temperature range from -20 to 40 degrees Celsius temperature = random.uniform(
-20, 40
) # Temperature range from -20 to 40 degrees Celsius
timestamp = time.time() # Current Unix timestamp timestamp = time.time() # Current Unix timestamp
return { return {
'location_id': location_id, "location_id": location_id,
'temperature': round(temperature, 2), "temperature": round(temperature, 2),
'timestamp': timestamp "timestamp": timestamp,
} }

28
Programming/PSD/zin3/python/code/generator/temperature_generator.py
View File

@ -1,26 +1,27 @@
import json import json
import random
import time
from confluent_kafka import Producer
import sys
import os import os
import random
import sys
import time
from confluent_kafka import Producer
# Add parent directory to path to import model # Add parent directory to path to import model
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from model.temperature_reading import TemperatureReading from model.temperature_reading import TemperatureReading
def delivery_report(err, msg): def delivery_report(err, msg):
"""Callback for message delivery reports""" """Callback for message delivery reports."""
if err is not None: if err is not None:
print(f'Message delivery failed: {err}') print(f"Message delivery failed: {err}")
else: else:
pass # Successfully delivered pass # Successfully delivered
def main(): def main():
# Set up Kafka producer with confluent-kafka # Set up Kafka producer with confluent-kafka
producer_config = { producer_config = {"bootstrap.servers": "localhost:9092"}
'bootstrap.servers': 'localhost:9092'
}
producer = Producer(producer_config) producer = Producer(producer_config)
# Generate a fixed number of thermometer IDs # Generate a fixed number of thermometer IDs
@ -36,19 +37,19 @@ def main():
reading = TemperatureReading( reading = TemperatureReading(
thermometer_id=thermometer_id, thermometer_id=thermometer_id,
timestamp=int(time.time() * 1000), timestamp=int(time.time() * 1000),
temperature=temperature temperature=temperature,
) )
# Convert to dictionary for JSON serialization # Convert to dictionary for JSON serialization
reading_dict = reading.to_dict() reading_dict = reading.to_dict()
payload = json.dumps(reading_dict).encode('utf-8') payload = json.dumps(reading_dict).encode("utf-8")
# Send to Kafka topic "Temperatura" # Send to Kafka topic "Temperatura"
producer.produce( producer.produce(
"Temperatura", "Temperatura",
key=thermometer_id.encode('utf-8'), key=thermometer_id.encode("utf-8"),
value=payload, value=payload,
callback=delivery_report callback=delivery_report,
) )
producer.flush(timeout=1) producer.flush(timeout=1)
# print(f"Sent: {json.dumps(reading_dict)}") # print(f"Sent: {json.dumps(reading_dict)}")
@ -62,5 +63,6 @@ def main():
# Wait for any outstanding messages to be delivered # Wait for any outstanding messages to be delivered
producer.flush() producer.flush()
if __name__ == "__main__": if __name__ == "__main__":
main() main()

4
Programming/PSD/zin3/python/code/model/temperature_alarm.py
View File

@ -8,7 +8,7 @@ class TemperatureAlarm:
return { return {
"thermometerId": self.thermometer_id, "thermometerId": self.thermometer_id,
"timestamp": self.timestamp, "timestamp": self.timestamp,
"temperature": self.temperature "temperature": self.temperature,
} }
@classmethod @classmethod
@ -16,5 +16,5 @@ class TemperatureAlarm:
return cls( return cls(
thermometer_id=data.get("thermometerId"), thermometer_id=data.get("thermometerId"),
timestamp=data.get("timestamp"), timestamp=data.get("timestamp"),
temperature=data.get("temperature") temperature=data.get("temperature"),
) )

4
Programming/PSD/zin3/python/code/model/temperature_reading.py
View File

@ -8,7 +8,7 @@ class TemperatureReading:
return { return {
"thermometerId": self.thermometer_id, "thermometerId": self.thermometer_id,
"timestamp": self.timestamp, "timestamp": self.timestamp,
"temperature": self.temperature "temperature": self.temperature,
} }
@classmethod @classmethod
@ -16,5 +16,5 @@ class TemperatureReading:
return cls( return cls(
thermometer_id=data.get("thermometerId"), thermometer_id=data.get("thermometerId"),
timestamp=data.get("timestamp"), timestamp=data.get("timestamp"),
temperature=data.get("temperature") temperature=data.get("temperature"),
) )

60
Programming/PSD/zin3/python/code/processor/temperature_anomaly_detector.py
View File

@ -1,23 +1,24 @@
import os
import json import json
import os
import sys import sys
from pyflink.datastream import StreamExecutionEnvironment
from pyflink.common.serialization import SimpleStringSchema from pyflink.common.serialization import SimpleStringSchema
from pyflink.datastream.connectors import FlinkKafkaConsumer, FlinkKafkaProducer
from pyflink.common.time import Time
from pyflink.common.typeinfo import Types from pyflink.common.typeinfo import Types
from pyflink.common.watermark_strategy import WatermarkStrategy from pyflink.common.watermark_strategy import WatermarkStrategy
from pyflink.datastream.functions import MapFunction, KeyedProcessFunction from pyflink.datastream import StreamExecutionEnvironment
from pyflink.datastream.state import ValueStateDescriptor from pyflink.datastream.connectors import FlinkKafkaConsumer, FlinkKafkaProducer
from pyflink.datastream.functions import KeyedProcessFunction, MapFunction
# Add parent directory to path to import model # Add parent directory to path to import model
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
class ReadingMapFunction(MapFunction): class ReadingMapFunction(MapFunction):
def map(self, value): def map(self, value):
data = json.loads(value) data = json.loads(value)
return (data["thermometerId"], data["timestamp"], data["temperature"]) return (data["thermometerId"], data["timestamp"], data["temperature"])
class AnomalyDetectionFunction(KeyedProcessFunction): class AnomalyDetectionFunction(KeyedProcessFunction):
def __init__(self): def __init__(self):
self.window_size = 10 * 1000 # 10 seconds in milliseconds self.window_size = 10 * 1000 # 10 seconds in milliseconds
@ -31,13 +32,13 @@ class AnomalyDetectionFunction(KeyedProcessFunction):
alarm = { alarm = {
"thermometerId": thermometer_id, "thermometerId": thermometer_id,
"timestamp": timestamp, "timestamp": timestamp,
"temperature": temperature "temperature": temperature,
} }
yield json.dumps(alarm) yield json.dumps(alarm)
def add_kafka_dependencies(env):
"""Add Kafka connector dependencies to the environment"""
def add_kafka_dependencies(env):
"""Add Kafka connector dependencies to the environment."""
# User-specific paths where JAR files are located # User-specific paths where JAR files are located
kafka_path = "/home/psd/Downloads/kafka_2.13-3.4.0/libs" kafka_path = "/home/psd/Downloads/kafka_2.13-3.4.0/libs"
flink_path = "/home/psd/Downloads/flink-1.17.0/lib" flink_path = "/home/psd/Downloads/flink-1.17.0/lib"
@ -45,7 +46,7 @@ def add_kafka_dependencies(env):
# Required JAR files # Required JAR files
kafka_jars = { kafka_jars = {
"kafka-clients": "kafka-clients-3.4.0.jar", "kafka-clients": "kafka-clients-3.4.0.jar",
"flink-connector-kafka": None # Will be detected or reported as missing "flink-connector-kafka": None, # Will be detected or reported as missing
} }
jar_paths = [] jar_paths = []
@ -73,9 +74,12 @@ def add_kafka_dependencies(env):
print(f"Place it in: {flink_path}") print(f"Place it in: {flink_path}")
if jar_paths: if jar_paths:
env.get_config().get_configuration().set_string("pipeline.jars", ";".join(jar_paths)) env.get_config().get_configuration().set_string(
"pipeline.jars", ";".join(jar_paths)
)
print(f"Added JAR files: {jar_paths}") print(f"Added JAR files: {jar_paths}")
def main(): def main():
# Create execution environment # Create execution environment
env = StreamExecutionEnvironment.get_execution_environment() env = StreamExecutionEnvironment.get_execution_environment()
@ -85,39 +89,36 @@ def main():
# Configure Kafka consumer # Configure Kafka consumer
consumer_props = { consumer_props = {
'bootstrap.servers': 'localhost:9092', "bootstrap.servers": "localhost:9092",
'group.id': 'temperature-anomaly-detector' "group.id": "temperature-anomaly-detector",
} }
# Create Kafka consumer # Create Kafka consumer
consumer = FlinkKafkaConsumer( consumer = FlinkKafkaConsumer(
topics=['Temperatura'], topics=["Temperatura"],
deserialization_schema=SimpleStringSchema(), deserialization_schema=SimpleStringSchema(),
properties=consumer_props properties=consumer_props,
) )
# Configure Kafka producer # Configure Kafka producer
producer_props = { producer_props = {"bootstrap.servers": "localhost:9092"}
'bootstrap.servers': 'localhost:9092'
}
# Create Kafka producer # Create Kafka producer
producer = FlinkKafkaProducer( producer = FlinkKafkaProducer(
topic='Alarm', topic="Alarm",
serialization_schema=SimpleStringSchema(), serialization_schema=SimpleStringSchema(),
producer_config=producer_props producer_config=producer_props,
) )
# Add source and transformations # Add source and transformations
env.add_source(consumer) \ env.add_source(consumer).map(
.map(ReadingMapFunction(), output_type=Types.TUPLE([Types.STRING(), Types.LONG(), Types.DOUBLE()])) \ ReadingMapFunction(),
.assign_timestamps_and_watermarks( output_type=Types.TUPLE([Types.STRING(), Types.LONG(), Types.DOUBLE()]),
WatermarkStrategy.for_monotonous_timestamps() ).assign_timestamps_and_watermarks(
.with_timestamp_assigner(lambda event, timestamp: event[1]) WatermarkStrategy.for_monotonous_timestamps().with_timestamp_assigner(
) \ lambda event, timestamp: event[1]
.key_by(lambda x: x[0]) \ )
.process(AnomalyDetectionFunction()) \ ).key_by(lambda x: x[0]).process(AnomalyDetectionFunction()).add_sink(producer)
.add_sink(producer)
# Print the execution plan # Print the execution plan
print(env.get_execution_plan()) print(env.get_execution_plan())
@ -125,5 +126,6 @@ def main():
# Execute # Execute
env.execute("Temperature Anomaly Detector with Time Windows") env.execute("Temperature Anomaly Detector with Time Windows")
if __name__ == "__main__": if __name__ == "__main__":
main() main()

34
Programming/PSD/zin3/python/code/visualizer/alarm_visualizer.py
View File

@ -1,17 +1,19 @@
import json import json
import os
import sys
import threading import threading
import tkinter as tk import tkinter as tk
from tkinter import ttk
from confluent_kafka import Consumer
import sys
import os
from datetime import datetime
from collections import Counter from collections import Counter
from datetime import datetime
from tkinter import ttk
from confluent_kafka import Consumer
# Add parent directory to path to import model # Add parent directory to path to import model
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from model.temperature_alarm import TemperatureAlarm from model.temperature_alarm import TemperatureAlarm
class AlarmVisualizer: class AlarmVisualizer:
def __init__(self, root): def __init__(self, root):
self.root = root self.root = root
@ -23,7 +25,9 @@ class AlarmVisualizer:
self.alarm_list.pack(fill=tk.BOTH, expand=True) self.alarm_list.pack(fill=tk.BOTH, expand=True)
# Create scrollbar # Create scrollbar
scrollbar = ttk.Scrollbar(self.alarm_list, orient="vertical", command=self.alarm_list.yview) scrollbar = ttk.Scrollbar(
self.alarm_list, orient="vertical", command=self.alarm_list.yview
)
scrollbar.pack(side=tk.RIGHT, fill=tk.Y) scrollbar.pack(side=tk.RIGHT, fill=tk.Y)
self.alarm_list.config(yscrollcommand=scrollbar.set) self.alarm_list.config(yscrollcommand=scrollbar.set)
@ -45,12 +49,12 @@ class AlarmVisualizer:
def consume_alarms(self): def consume_alarms(self):
# Set up Kafka consumer with confluent-kafka # Set up Kafka consumer with confluent-kafka
consumer_config = { consumer_config = {
'bootstrap.servers': 'localhost:9092', "bootstrap.servers": "localhost:9092",
'group.id': 'alarm-visualizer', "group.id": "alarm-visualizer",
'auto.offset.reset': 'latest' "auto.offset.reset": "latest",
} }
consumer = Consumer(consumer_config) consumer = Consumer(consumer_config)
consumer.subscribe(['Alarm']) consumer.subscribe(["Alarm"])
try: try:
while True: while True:
@ -62,12 +66,12 @@ class AlarmVisualizer:
continue continue
# Parse the alarm message # Parse the alarm message
alarm_data = json.loads(msg.value().decode('utf-8')) alarm_data = json.loads(msg.value().decode("utf-8"))
alarm = TemperatureAlarm.from_dict(alarm_data) alarm = TemperatureAlarm.from_dict(alarm_data)
# Format the timestamp # Format the timestamp
timestamp_dt = datetime.fromtimestamp(alarm.timestamp / 1000) timestamp_dt = datetime.fromtimestamp(alarm.timestamp / 1000)
formatted_date = timestamp_dt.strftime('%Y-%m-%d %H:%M:%S') formatted_date = timestamp_dt.strftime("%Y-%m-%d %H:%M:%S")
# Create alarm message # Create alarm message
alarm_message = f"⚠️ ALARM: Thermometer {alarm.thermometer_id} reported {alarm.temperature:.2f}°C at {formatted_date}" alarm_message = f"⚠️ ALARM: Thermometer {alarm.thermometer_id} reported {alarm.temperature:.2f}°C at {formatted_date}"
@ -93,14 +97,18 @@ class AlarmVisualizer:
# Format stats string # Format stats string
if self.thermometer_alarm_count: if self.thermometer_alarm_count:
stats_parts = [f"{id}={count}" for id, count in self.thermometer_alarm_count.items()] stats_parts = [
f"{id}={count}" for id, count in self.thermometer_alarm_count.items()
]
stats_text = "Alarm Counts: " + ", ".join(stats_parts) stats_text = "Alarm Counts: " + ", ".join(stats_parts)
self.stats_label.config(text=stats_text) self.stats_label.config(text=stats_text)
def main(): def main():
root = tk.Tk() root = tk.Tk()
app = AlarmVisualizer(root) app = AlarmVisualizer(root)
root.mainloop() root.mainloop()
if __name__ == "__main__": if __name__ == "__main__":
main() main()

3
Programming/TRAK/.gitignore vendored
View File

@ -168,3 +168,6 @@ cython_debug/
# *.exr -- the environment map below is tracked; see root .gitignore # *.exr -- the environment map below is tracked; see root .gitignore
!*.exr !*.exr
# Output of `../../run.sh run`, which must never land on a tracked
# deliverable in outputs/.
build/

59
Programming/TRAK/code/mapa_srodowiska/main.py
View File

@ -1,10 +1,10 @@
import OpenEXR
import Imath
import os import os
import Imath
import OpenEXR
import OpenGL.GL as gl import OpenGL.GL as gl
import OpenGL.GLUT as glut
import OpenGL.GLU as glu import OpenGL.GLU as glu
import math import OpenGL.GLUT as glut
# Camera parameters # Camera parameters
camera_angle_x = 0.0 camera_angle_x = 0.0
@ -14,15 +14,16 @@ mouse_last_x = 0
mouse_last_y = 0 mouse_last_y = 0
mouse_left_down = False mouse_left_down = False
def load_hdr_environment_map(filepath): def load_hdr_environment_map(filepath):
""" """Load an HDR environment map from an OpenEXR file.
Load an HDR environment map from an OpenEXR file.
Args: Args:
filepath (str): Path to the HDR file. filepath (str): Path to the HDR file.
Returns: Returns:
tuple: A tuple containing the width, height, and a bytes object representing the HDR environment map in RGB format. tuple: A tuple containing the width, height, and a bytes object representing the HDR environment map in RGB format.
""" """
if not os.path.exists(filepath): if not os.path.exists(filepath):
raise FileNotFoundError(f"File not found: {filepath}") raise FileNotFoundError(f"File not found: {filepath}")
@ -35,16 +36,16 @@ def load_hdr_environment_map(filepath):
try: try:
exr_file = OpenEXR.InputFile(filepath) exr_file = OpenEXR.InputFile(filepath)
except Exception as e: except Exception as e:
raise OSError(f"Unable to open '{filepath}' for read: {str(e)}") raise OSError(f"Unable to open '{filepath}' for read: {e!s}")
# Get the image dimensions # Get the image dimensions
header = exr_file.header() header = exr_file.header()
dw = header['dataWindow'] dw = header["dataWindow"]
width = dw.max.x - dw.min.x + 1 width = dw.max.x - dw.min.x + 1
height = dw.max.y - dw.min.y + 1 height = dw.max.y - dw.min.y + 1
# Define the channel names (R, G, B) # Define the channel names (R, G, B)
channels = ['R', 'G', 'B'] channels = ["R", "G", "B"]
# Read the channel data # Read the channel data
channel_data = { channel_data = {
@ -59,7 +60,9 @@ def load_hdr_environment_map(filepath):
for j in range(height): for j in range(height):
for k in range(width): for k in range(width):
index = (j * width + k) * 3 * 4 + i * 4 index = (j * width + k) * 3 * 4 + i * 4
hdr_image[index:index + 4] = channel_buffer[(j * width + k) * 4:(j * width + k + 1) * 4] hdr_image[index : index + 4] = channel_buffer[
(j * width + k) * 4 : (j * width + k + 1) * 4
]
# Flip the image vertically # Flip the image vertically
flipped_hdr_image = bytearray(width * height * 3 * 4) flipped_hdr_image = bytearray(width * height * 3 * 4)
@ -67,10 +70,13 @@ def load_hdr_environment_map(filepath):
for j in range(height): for j in range(height):
src_index = j * row_size src_index = j * row_size
dst_index = (height - 1 - j) * row_size dst_index = (height - 1 - j) * row_size
flipped_hdr_image[dst_index:dst_index + row_size] = hdr_image[src_index:src_index + row_size] flipped_hdr_image[dst_index : dst_index + row_size] = hdr_image[
src_index : src_index + row_size
]
return width, height, bytes(flipped_hdr_image) return width, height, bytes(flipped_hdr_image)
def display_hdr_image(width, height, hdr_image): def display_hdr_image(width, height, hdr_image):
gl.glClear(gl.GL_COLOR_BUFFER_BIT | gl.GL_DEPTH_BUFFER_BIT) gl.glClear(gl.GL_COLOR_BUFFER_BIT | gl.GL_DEPTH_BUFFER_BIT)
@ -86,13 +92,34 @@ def display_hdr_image(width, height, hdr_image):
gl.glTexParameteri(gl.GL_TEXTURE_2D, gl.GL_TEXTURE_MAG_FILTER, gl.GL_LINEAR) gl.glTexParameteri(gl.GL_TEXTURE_2D, gl.GL_TEXTURE_MAG_FILTER, gl.GL_LINEAR)
# Load the texture # Load the texture
gl.glTexImage2D(gl.GL_TEXTURE_2D, 0, gl.GL_RGB32F, width, height, 0, gl.GL_RGB, gl.GL_FLOAT, hdr_image) gl.glTexImage2D(
gl.GL_TEXTURE_2D,
0,
gl.GL_RGB32F,
width,
height,
0,
gl.GL_RGB,
gl.GL_FLOAT,
hdr_image,
)
# Set up the viewport and projection # Set up the viewport and projection
gl.glViewport(0, 0, glut.glutGet(glut.GLUT_WINDOW_WIDTH), glut.glutGet(glut.GLUT_WINDOW_HEIGHT)) gl.glViewport(
0,
0,
glut.glutGet(glut.GLUT_WINDOW_WIDTH),
glut.glutGet(glut.GLUT_WINDOW_HEIGHT),
)
gl.glMatrixMode(gl.GL_PROJECTION) gl.glMatrixMode(gl.GL_PROJECTION)
gl.glLoadIdentity() gl.glLoadIdentity()
glu.gluPerspective(45.0, glut.glutGet(glut.GLUT_WINDOW_WIDTH) / float(glut.glutGet(glut.GLUT_WINDOW_HEIGHT)), 0.1, 100.0) glu.gluPerspective(
45.0,
glut.glutGet(glut.GLUT_WINDOW_WIDTH)
/ float(glut.glutGet(glut.GLUT_WINDOW_HEIGHT)),
0.1,
100.0,
)
gl.glMatrixMode(gl.GL_MODELVIEW) gl.glMatrixMode(gl.GL_MODELVIEW)
gl.glLoadIdentity() gl.glLoadIdentity()
@ -112,6 +139,7 @@ def display_hdr_image(width, height, hdr_image):
glut.glutSwapBuffers() glut.glutSwapBuffers()
def mouse_motion(x, y): def mouse_motion(x, y):
global mouse_last_x, mouse_last_y, camera_angle_x, camera_angle_y global mouse_last_x, mouse_last_y, camera_angle_x, camera_angle_y
if mouse_left_down: if mouse_left_down:
@ -123,6 +151,7 @@ def mouse_motion(x, y):
mouse_last_y = y mouse_last_y = y
glut.glutPostRedisplay() glut.glutPostRedisplay()
def mouse_button(button, state, x, y): def mouse_button(button, state, x, y):
global mouse_left_down global mouse_left_down
if button == glut.GLUT_LEFT_BUTTON: if button == glut.GLUT_LEFT_BUTTON:
@ -133,6 +162,7 @@ def mouse_button(button, state, x, y):
elif state == glut.GLUT_UP: elif state == glut.GLUT_UP:
mouse_left_down = False mouse_left_down = False
def main(filepath): def main(filepath):
try: try:
width, height, hdr_map = load_hdr_environment_map(filepath) width, height, hdr_map = load_hdr_environment_map(filepath)
@ -156,6 +186,7 @@ def main(filepath):
except Exception as e: except Exception as e:
print(e) print(e)
# Example usage # Example usage
if __name__ == "__main__": if __name__ == "__main__":
filepath = "lilienstein_4k.exr" filepath = "lilienstein_4k.exr"

81
Programming/TRAK/code/photonmapping/python/main.py
View File

@ -1,9 +1,11 @@
import numpy as np
import matplotlib.pyplot as plt
import time import time
import matplotlib.pyplot as plt
import numpy as np
PHOTONS = 0 PHOTONS = 0
# Define basic vector operations # Define basic vector operations
class Vector3: class Vector3:
def __init__(self, x, y, z): def __init__(self, x, y, z):
@ -12,23 +14,24 @@ class Vector3:
self.z = z self.z = z
def __add__(self, other): def __add__(self, other):
return Vector3(self.x+other.x, self.y+other.y, self.z+other.z) return Vector3(self.x + other.x, self.y + other.y, self.z + other.z)
def __sub__(self, other): def __sub__(self, other):
return Vector3(self.x-other.x, self.y-other.y, self.z-other.z) return Vector3(self.x - other.x, self.y - other.y, self.z - other.z)
def __mul__(self, scalar): def __mul__(self, scalar):
return Vector3(self.x*scalar, self.y*scalar, self.z*scalar) return Vector3(self.x * scalar, self.y * scalar, self.z * scalar)
def dot(self, other): def dot(self, other):
return self.x*other.x + self.y*other.y + self.z*other.z return self.x * other.x + self.y * other.y + self.z * other.z
def norm(self): def norm(self):
return np.sqrt(self.dot(self)) return np.sqrt(self.dot(self))
def normalize(self): def normalize(self):
n = self.norm() n = self.norm()
return Vector3(self.x/n, self.y/n, self.z/n) return Vector3(self.x / n, self.y / n, self.z / n)
# Define the photon # Define the photon
class Photon: class Photon:
@ -37,6 +40,7 @@ class Photon:
self.direction = direction self.direction = direction
self.power = power self.power = power
# Define a simple sphere # Define a simple sphere
class Sphere: class Sphere:
def __init__(self, center, radius, color): def __init__(self, center, radius, color):
@ -49,19 +53,19 @@ class Sphere:
oc = ray_origin - self.center oc = ray_origin - self.center
a = ray_direction.dot(ray_direction) a = ray_direction.dot(ray_direction)
b = 2.0 * oc.dot(ray_direction) b = 2.0 * oc.dot(ray_direction)
c = oc.dot(oc) - self.radius*self.radius c = oc.dot(oc) - self.radius * self.radius
discriminant = b*b - 4*a*c discriminant = b * b - 4 * a * c
if discriminant < 0: if discriminant < 0:
return None # No intersection return None # No intersection
else: t = (-b - np.sqrt(discriminant)) / (2.0 * a)
t = (-b - np.sqrt(discriminant)) / (2.0*a) if t < 0:
if t < 0: t = (-b + np.sqrt(discriminant)) / (2.0 * a)
t = (-b + np.sqrt(discriminant)) / (2.0*a) if t < 0:
if t < 0: return None
return None hit_point = ray_origin + ray_direction * t
hit_point = ray_origin + ray_direction * t normal = (hit_point - self.center).normalize()
normal = (hit_point - self.center).normalize() return (t, hit_point, normal)
return (t, hit_point, normal)
# Define a simple plane # Define a simple plane
class Plane: class Plane:
@ -79,9 +83,12 @@ class Plane:
return (t, hit_point, self.normal) return (t, hit_point, self.normal)
return None return None
# Scene setup # Scene setup
sphere = Sphere(Vector3(0, 0, -5), 1.0, np.array([1, 0, 0])) # Red sphere sphere = Sphere(Vector3(0, 0, -5), 1.0, np.array([1, 0, 0])) # Red sphere
plane = Plane(Vector3(0, -1, 0), Vector3(0, 1, 0), np.array([0.5, 0.5, 0.5])) # Gray plane plane = Plane(
Vector3(0, -1, 0), Vector3(0, 1, 0), np.array([0.5, 0.5, 0.5])
) # Gray plane
objects = [sphere, plane] objects = [sphere, plane]
@ -97,6 +104,7 @@ num_photons = 10000 # Number of photons to emit
max_depth = 5 # Maximum number of bounces max_depth = 5 # Maximum number of bounces
gather_radius = 0.5 # Radius for radiance estimation gather_radius = 0.5 # Radius for radiance estimation
def emit_photons(): def emit_photons():
for _ in range(num_photons): for _ in range(num_photons):
# Emit photons in random directions from the light source # Emit photons in random directions from the light source
@ -105,6 +113,7 @@ def emit_photons():
photon = Photon(light_position, direction, power) photon = Photon(light_position, direction, power)
trace_photon(photon, 0) trace_photon(photon, 0)
def trace_photon(photon, depth): def trace_photon(photon, depth):
global PHOTONS global PHOTONS
PHOTONS += 1 PHOTONS += 1
@ -131,20 +140,23 @@ def trace_photon(photon, depth):
photon.direction = new_direction photon.direction = new_direction
# Absorb some power # Absorb some power
photon.power = photon.power * 0.8 # Simple absorption photon.power = photon.power * 0.8 # Simple absorption
trace_photon(photon, depth+1) trace_photon(photon, depth + 1)
def random_unit_vector(): def random_unit_vector():
theta = np.random.uniform(0, 2*np.pi) theta = np.random.uniform(0, 2 * np.pi)
z = np.random.uniform(-1, 1) z = np.random.uniform(-1, 1)
r = np.sqrt(1 - z*z) r = np.sqrt(1 - z * z)
return Vector3(r * np.cos(theta), r * np.sin(theta), z) return Vector3(r * np.cos(theta), r * np.sin(theta), z)
def random_hemisphere_direction(normal): def random_hemisphere_direction(normal):
dir = random_unit_vector() dir = random_unit_vector()
if dir.dot(normal) < 0: if dir.dot(normal) < 0:
dir = Vector3(-dir.x, -dir.y, -dir.z) dir = Vector3(-dir.x, -dir.y, -dir.z)
return dir return dir
def render_image(width, height): def render_image(width, height):
aspect_ratio = width / height aspect_ratio = width / height
fov = np.pi / 3 # 60 degrees field of view fov = np.pi / 3 # 60 degrees field of view
@ -160,6 +172,7 @@ def render_image(width, height):
image[y, x, :] = np.clip(color, 0, 1) image[y, x, :] = np.clip(color, 0, 1)
return image return image
def trace_ray(ray_origin, ray_direction): def trace_ray(ray_origin, ray_direction):
closest_t = np.inf closest_t = np.inf
hit_object = None hit_object = None
@ -178,8 +191,8 @@ def trace_ray(ray_origin, ray_direction):
direct_light = compute_direct_light(hit_point, normal) direct_light = compute_direct_light(hit_point, normal)
indirect_light = estimate_radiance(hit_point, normal) indirect_light = estimate_radiance(hit_point, normal)
return color * (direct_light + indirect_light) return color * (direct_light + indirect_light)
else: return np.array([0, 0, 0]) # Background color
return np.array([0, 0, 0]) # Background color
def compute_direct_light(point, normal): def compute_direct_light(point, normal):
# Simple Lambertian reflection from light source # Simple Lambertian reflection from light source
@ -195,9 +208,9 @@ def compute_direct_light(point, normal):
break break
if in_shadow: if in_shadow:
return np.array([0, 0, 0]) return np.array([0, 0, 0])
else: intensity = max(0, normal.dot(direction_to_light))
intensity = max(0, normal.dot(direction_to_light)) return intensity * light_power / (4 * np.pi * (light_position - point).norm() ** 2)
return intensity * light_power / (4 * np.pi * (light_position - point).norm()**2)
def estimate_radiance(point, normal): def estimate_radiance(point, normal):
# Gather photons within the gather_radius # Gather photons within the gather_radius
@ -207,11 +220,12 @@ def estimate_radiance(point, normal):
if distance < gather_radius: if distance < gather_radius:
weight = max(0, normal.dot((photon_pos - point).normalize())) weight = max(0, normal.dot((photon_pos - point).normalize()))
accumulated_power += photon_power * weight accumulated_power += photon_power * weight
area = np.pi * gather_radius ** 2 area = np.pi * gather_radius**2
return accumulated_power / (area * num_photons) return accumulated_power / (area * num_photons)
# Main execution # Main execution
if __name__ == '__main__': if __name__ == "__main__":
print("Emitting photons...") print("Emitting photons...")
emit_photons() emit_photons()
@ -221,15 +235,16 @@ if __name__ == '__main__':
t0 = time.time() t0 = time.time()
image = render_image(width, height) image = render_image(width, height)
print (f"Render Took: {round(time.time() - t0, 2)}s\n" print(
f"Render Took: {round(time.time() - t0, 2)}s\n"
f"resolution: {width}x{height}\n" f"resolution: {width}x{height}\n"
f"photons (emitted): {num_photons}\n" f"photons (emitted): {num_photons}\n"
f"photons (reflected): {PHOTONS - num_photons}\n" f"photons (reflected): {PHOTONS - num_photons}\n"
f"photons (total): {PHOTONS}\n" f"photons (total): {PHOTONS}\n"
f"rays: {width*height}" f"rays: {width * height}"
) )
# Display the image # Display the image
plt.imshow(image) plt.imshow(image)
plt.axis('off') plt.axis("off")
plt.show() plt.show()

6
Programming/TRAK/lab6.py
View File

@ -1,4 +1,2 @@
import bpy script_path = "" # path to script
import os exec(compile(open(script_path).read(), script_path, "exec"))
script_path = '' # path to script
exec(compile(open(script_path).read(), script_path, 'exec'))

135
Programming/TRAK/main.py
View File

@ -1,33 +1,96 @@
import argparse import argparse
from rendering import ray_trace
from utils import load_config, parse_resolution
import importlib import importlib
import os import os
import matplotlib.pyplot as plt import matplotlib.pyplot as plt
from photon_mapping import render_photon_mapping from photon_mapping import render_photon_mapping
from rendering import ray_trace
from utils import load_config, parse_resolution
def main(): def main():
# default config # default config
config = load_config('config.ini') config = load_config("config.ini")
# Parse # Parse
parser = argparse.ArgumentParser(description="Rendering Program") parser = argparse.ArgumentParser(description="Rendering Program")
parser.add_argument('--algorithm', type=str, help='Algorithm to use', default=config.get('DEFAULT', 'algorithm')) parser.add_argument(
parser.add_argument('--scene', type=str, help='Name of the scene to render (without .py).', default=config.get('DEFAULT', 'scene')) "--algorithm",
parser.add_argument('--environment', type=str, help='Environment file', default=config.get('DEFAULT', 'environment')) type=str,
parser.add_argument('--env_blur', type=str, help='Environment blur', default=config.get('DEFAULT', 'env_blur')) help="Algorithm to use",
parser.add_argument('--resolution', type=str, help='Image resolution (WIDTHxHEIGHT)', default=config.get("DEFAULT", "algorithm"),
default=config.get('DEFAULT', 'resolution')) )
parser.add_argument("--samples_per_pixel", type=int, default=config.get('ray_tracing', 'samples_per_pixel'), help="Samples per pixel for rendering.") parser.add_argument(
parser.add_argument("--output", type=str, default=config.get('DEFAULT', 'output'), help="Output file name.") "--scene",
type=str,
parser.add_argument('--num_spheres', type=int, default=3, help='Number of spheres in the scene for Ray Tracing 0') help="Name of the scene to render (without .py).",
parser.add_argument('--num_photons', type=int, default=config.getint('photon_mapping', 'num_photons'), help='Number of photons for photon mapping') default=config.get("DEFAULT", "scene"),
parser.add_argument('--max_depth', type=int, default=config.getint('photon_mapping', 'max_depth'), )
help='Maximum depth for photon tracing') parser.add_argument(
parser.add_argument('--gather_radius', type=float, default=config.getfloat('photon_mapping', 'gather_radius'), "--environment",
help='Radius for radiance estimation in photon mapping') type=str,
help="Environment file",
default=config.get("DEFAULT", "environment"),
)
parser.add_argument(
"--env_blur",
type=str,
help="Environment blur",
default=config.get("DEFAULT", "env_blur"),
)
parser.add_argument(
"--resolution",
type=str,
help="Image resolution (WIDTHxHEIGHT)",
default=config.get("DEFAULT", "resolution"),
)
parser.add_argument(
"--samples_per_pixel",
type=int,
default=config.get("ray_tracing", "samples_per_pixel"),
help="Samples per pixel for rendering.",
)
parser.add_argument(
"--output",
type=str,
default=config.get("DEFAULT", "output"),
help="Output file name.",
)
parser.add_argument(
"--num_spheres",
type=int,
default=3,
help="Number of spheres in the scene for Ray Tracing 0",
)
parser.add_argument(
"--num_photons",
type=int,
default=config.getint("photon_mapping", "num_photons"),
help="Number of photons for photon mapping",
)
parser.add_argument(
"--max_depth",
type=int,
default=config.getint("photon_mapping", "max_depth"),
help="Maximum depth for photon tracing",
)
parser.add_argument(
"--gather_radius",
type=float,
default=config.getfloat("photon_mapping", "gather_radius"),
help="Radius for radiance estimation in photon mapping",
)
# The render is always written to outputs/; this only controls whether a
# viewer is ALSO popped up. It defaults off because img.show() shells out
# to xdg-open and then blocks until a human closes the window -- which
# left ../../run.sh run Programming/TRAK hanging for 45 minutes after the
# ray trace had actually finished.
parser.add_argument(
"--show",
action="store_true",
help="also open the finished image in a viewer (blocks until closed)",
)
args = parser.parse_args() args = parser.parse_args()
@ -37,8 +100,13 @@ def main():
if args.algorithm == "ray_tracing0": if args.algorithm == "ray_tracing0":
print("Starting ray tracing zero...") print("Starting ray tracing zero...")
# ray_trace(args.scene, args.environment_map, image_width=width, image_height=height, output_file="output_ray_traced.png") # ray_trace(args.scene, args.environment_map, image_width=width, image_height=height, output_file="output_ray_traced.png")
ray_trace(args.num_spheres, args.environment_map, image_width=width, image_height=height, # na razie generujemy w kodzie, ale potem trzeba będzie obj wczytywać ray_trace(
output_file="output_ray_traced.png") args.num_spheres,
args.environment_map,
image_width=width,
image_height=height, # na razie generujemy w kodzie, ale potem trzeba będzie obj wczytywać
output_file="output_ray_traced.png",
)
elif args.algorithm == "ray_tracing": elif args.algorithm == "ray_tracing":
print("Starting ray tracing...") print("Starting ray tracing...")
try: try:
@ -48,29 +116,40 @@ def main():
print(f"Error: Scene '{args.scene}' not found in the 'scenes' directory.") print(f"Error: Scene '{args.scene}' not found in the 'scenes' directory.")
return return
try: try:
scene = scene_module.setup_scene(width=width, height=height, environment=f"{args.environment}") scene = scene_module.setup_scene(
width=width, height=height, environment=f"{args.environment}"
)
except AttributeError: except AttributeError:
print(f"Error: Scene '{args.scene}' does not define a `setup_scene` function.") print(
f"Error: Scene '{args.scene}' does not define a `setup_scene` function."
)
return return
# Renderowanie # Renderowanie
print(f"Rendering scene '{args.scene}' with {args.samples_per_pixel} samples per pixel...") print(
f"Rendering scene '{args.scene}' with {args.samples_per_pixel} samples per pixel..."
)
img = scene.render(samples_per_pixel=args.samples_per_pixel) img = scene.render(samples_per_pixel=args.samples_per_pixel)
output_path = os.path.join("outputs", args.output) output_path = os.path.join("outputs", args.output)
img.save(output_path) img.save(output_path)
print(f"Image saved to {output_path}") print(f"Image saved to {output_path}")
img.show() if args.show:
img.show()
elif args.algorithm == "photon_mapping": elif args.algorithm == "photon_mapping":
print("Starting photon mapping...") print("Starting photon mapping...")
image = render_photon_mapping(width, height, args.num_photons, args.max_depth, args.gather_radius) image = render_photon_mapping(
width, height, args.num_photons, args.max_depth, args.gather_radius
)
plt.imshow(image) plt.imshow(image)
plt.axis('off') plt.axis("off")
output_path = os.path.join("outputs", args.output) output_path = os.path.join("outputs", args.output)
plt.savefig(output_path) plt.savefig(output_path)
print(f"Image saved to {output_path}") print(f"Image saved to {output_path}")
plt.show() if args.show:
plt.show()
else: else:
print(f"Unknown algorithm: {args.algorithm}") print(f"Unknown algorithm: {args.algorithm}")
return return
if __name__ == '__main__':
if __name__ == "__main__":
main() main()

35
Programming/TRAK/photon_mapping.py
View File

@ -1,5 +1,5 @@
import numpy as np import numpy as np
import matplotlib.pyplot as plt
# Define basic vector operations # Define basic vector operations
class Vector3: class Vector3:
@ -49,15 +49,14 @@ class Sphere:
discriminant = b * b - 4 * a * c discriminant = b * b - 4 * a * c
if discriminant < 0: if discriminant < 0:
return None return None
else: t = (-b - np.sqrt(discriminant)) / (2.0 * a)
t = (-b - np.sqrt(discriminant)) / (2.0 * a) if t < 0:
if t < 0: t = (-b + np.sqrt(discriminant)) / (2.0 * a)
t = (-b + np.sqrt(discriminant)) / (2.0 * a) if t < 0:
if t < 0: return None
return None hit_point = ray_origin + ray_direction * t
hit_point = ray_origin + ray_direction * t normal = (hit_point - self.center).normalize()
normal = (hit_point - self.center).normalize() return (t, hit_point, normal)
return (t, hit_point, normal)
class Plane: class Plane:
@ -80,7 +79,9 @@ def render_photon_mapping(width, height, num_photons, max_depth, gather_radius):
# Photon mapping logic # Photon mapping logic
photon_map = [] photon_map = []
sphere = Sphere(Vector3(0, 0, -5), 1.0, np.array([1, 0, 0])) # Red sphere sphere = Sphere(Vector3(0, 0, -5), 1.0, np.array([1, 0, 0])) # Red sphere
plane = Plane(Vector3(0, -1, 0), Vector3(0, 1, 0), np.array([0.5, 0.5, 0.5])) # Gray plane plane = Plane(
Vector3(0, -1, 0), Vector3(0, 1, 0), np.array([0.5, 0.5, 0.5])
) # Gray plane
objects = [sphere, plane] objects = [sphere, plane]
light_position = Vector3(-5, 5, -5) light_position = Vector3(-5, 5, -5)
@ -145,8 +146,7 @@ def render_photon_mapping(width, height, num_photons, max_depth, gather_radius):
direct_light = compute_direct_light(hit_point, normal) direct_light = compute_direct_light(hit_point, normal)
indirect_light = estimate_radiance(hit_point, normal) indirect_light = estimate_radiance(hit_point, normal)
return color * (direct_light + indirect_light) return color * (direct_light + indirect_light)
else: return np.array([0, 0, 0])
return np.array([0, 0, 0])
def compute_direct_light(point, normal): def compute_direct_light(point, normal):
direction_to_light = (light_position - point).normalize() direction_to_light = (light_position - point).normalize()
@ -160,9 +160,10 @@ def render_photon_mapping(width, height, num_photons, max_depth, gather_radius):
break break
if in_shadow: if in_shadow:
return np.array([0, 0, 0]) return np.array([0, 0, 0])
else: intensity = max(0, normal.dot(direction_to_light))
intensity = max(0, normal.dot(direction_to_light)) return (
return intensity * light_power / (4 * np.pi * (light_position - point).norm() ** 2) intensity * light_power / (4 * np.pi * (light_position - point).norm() ** 2)
)
def estimate_radiance(point, normal): def estimate_radiance(point, normal):
accumulated_power = np.array([0.0, 0.0, 0.0]) accumulated_power = np.array([0.0, 0.0, 0.0])
@ -171,7 +172,7 @@ def render_photon_mapping(width, height, num_photons, max_depth, gather_radius):
if distance < gather_radius: if distance < gather_radius:
weight = max(0, normal.dot((photon_pos - point).normalize())) weight = max(0, normal.dot((photon_pos - point).normalize()))
accumulated_power += photon_power * weight accumulated_power += photon_power * weight
area = np.pi * gather_radius ** 2 area = np.pi * gather_radius**2
return accumulated_power / area return accumulated_power / area
emit_photons() emit_photons()

301
Programming/TRAK/rendering.py
View File

@ -1,33 +1,38 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
""" Renders an image using raytracing """ """Renders an image using raytracing."""
import numpy as np
import matplotlib.pyplot as plt
import time import time
def ray_trace(num_spheres, environment, image_width=400, image_height=300, output_file="fig.png"): import matplotlib.pyplot as plt
import numpy as np
def ray_trace(
num_spheres, environment, image_width=400, image_height=300, output_file="fig.png"
):
IMAGE_WIDTH = image_width IMAGE_WIDTH = image_width
IMAGE_HEIGHT = image_height IMAGE_HEIGHT = image_height
def normalize(vector): def normalize(vector):
""" """Normalize a vector.
Normalize a vector.
Parameters: Parameters
----------
vector (numpy.ndarray): The input vector to be normalized. vector (numpy.ndarray): The input vector to be normalized.
Returns: Returns
-------
numpy.ndarray: The normalized vector. numpy.ndarray: The normalized vector.
""" """
vector /= np.linalg.norm(vector) vector /= np.linalg.norm(vector)
return vector return vector
def intersect_plane(ray_origin, ray_direction, plane_point, plane_normal): def intersect_plane(ray_origin, ray_direction, plane_point, plane_normal):
""" """Calculate the intersection of a ray with a plane.
Calculate the intersection of a ray with a plane.
Parameters: Parameters
----------
ray_origin (numpy.ndarray): A 3D point representing the origin of the ray. ray_origin (numpy.ndarray): A 3D point representing the origin of the ray.
ray_direction (numpy.ndarray): A normalized 3D vector representing the ray_direction (numpy.ndarray): A normalized 3D vector representing the
direction of the ray. direction of the ray.
@ -35,11 +40,13 @@ def ray_trace(num_spheres, environment, image_width=400, image_height=300, outpu
plane_normal (numpy.ndarray): A normalized 3D vector representing plane_normal (numpy.ndarray): A normalized 3D vector representing
the normal of the plane. the normal of the plane.
Returns: Returns
-------
float: The distance from the origin ray_origin to the intersection float: The distance from the origin ray_origin to the intersection
point with the plane. point with the plane.
Returns +inf if there is no intersection or if the intersection is Returns +inf if there is no intersection or if the intersection is
behind the origin. behind the origin.
""" """
denom = np.dot(ray_direction, plane_normal) denom = np.dot(ray_direction, plane_normal)
if np.abs(denom) < 1e-6: if np.abs(denom) < 1e-6:
@ -49,12 +56,11 @@ def ray_trace(num_spheres, environment, image_width=400, image_height=300, outpu
return np.inf return np.inf
return d return d
def intersect_sphere(ray_origin, ray_direction, sphere_center, sphere_radius): def intersect_sphere(ray_origin, ray_direction, sphere_center, sphere_radius):
""" """Calculate the intersection of a ray with a sphere.
Calculate the intersection of a ray with a sphere.
Parameters: Parameters
----------
ray_origin (numpy.ndarray): A 3D point representing the origin of the ray. ray_origin (numpy.ndarray): A 3D point representing the origin of the ray.
ray_direction (numpy.ndarray): A normalized 3D vector representing the ray_direction (numpy.ndarray): A normalized 3D vector representing the
direction of the ray. direction of the ray.
@ -62,11 +68,13 @@ def ray_trace(num_spheres, environment, image_width=400, image_height=300, outpu
the center of the sphere. the center of the sphere.
sphere_radius (float): The radius of the sphere. sphere_radius (float): The radius of the sphere.
Returns: Returns
-------
float: The distance from the origin ray_origin to the intersection float: The distance from the origin ray_origin to the intersection
point with the sphere. point with the sphere.
Returns +inf if there is no intersection or if the intersection is Returns +inf if there is no intersection or if the intersection is
behind the origin. behind the origin.
""" """
a = np.dot(ray_direction, ray_direction) a = np.dot(ray_direction, ray_direction)
origin_to_center = ray_origin - sphere_center origin_to_center = ray_origin - sphere_center
@ -76,23 +84,24 @@ def ray_trace(num_spheres, environment, image_width=400, image_height=300, outpu
disc = b * b - 4 * a * c disc = b * b - 4 * a * c
return calculate_sphere_intersection(a, b, c, disc) return calculate_sphere_intersection(a, b, c, disc)
def calculate_sphere_intersection(a, b, c, disc): def calculate_sphere_intersection(a, b, c, disc):
""" """Calculate the
Calculate the
intersection distance of a ray with a sphere using the quadratic formula. intersection distance of a ray with a sphere using the quadratic formula.
Parameters: Parameters
----------
a (float): Coefficient of t^2 in the quadratic equation. a (float): Coefficient of t^2 in the quadratic equation.
b (float): Coefficient of t in the quadratic equation. b (float): Coefficient of t in the quadratic equation.
c (float): Constant term in the quadratic equation. c (float): Constant term in the quadratic equation.
disc (float): Discriminant of the quadratic equation. disc (float): Discriminant of the quadratic equation.
Returns: Returns
-------
float: float:
The distance from the origin to the intersection point with the sphere. The distance from the origin to the intersection point with the sphere.
Returns +inf if there is no intersection Returns +inf if there is no intersection
or if the intersection is behind the origin. or if the intersection is behind the origin.
""" """
if disc > 0: if disc > 0:
distance_squared = np.sqrt(disc) distance_squared = np.sqrt(disc)
@ -108,112 +117,121 @@ def ray_trace(num_spheres, environment, image_width=400, image_height=300, outpu
return t1 if t0 < 0 else t0 return t1 if t0 < 0 else t0
return np.inf return np.inf
def intersect(ray_origin, ray_direction, object_): def intersect(ray_origin, ray_direction, object_):
""" """Calculate the intersection of a ray with an object.
Calculate the intersection of a ray with an object.
Parameters: Parameters
----------
ray_origin (numpy.ndarray): A 3D point representing the origin of the ray. ray_origin (numpy.ndarray): A 3D point representing the origin of the ray.
ray_direction (numpy.ndarray): A normalized 3D vector representing the ray_direction (numpy.ndarray): A normalized 3D vector representing the
direction of the ray. direction of the ray.
obj (dict): A dictionary representing the object with keys obj (dict): A dictionary representing the object with keys
'type', 'position', 'normal' (for planes), and 'radius' (for spheres). 'type', 'position', 'normal' (for planes), and 'radius' (for spheres).
Returns: Returns
-------
float: The distance from the origin ray_origin to the intersection float: The distance from the origin ray_origin to the intersection
point with the object. point with the object.
Returns +inf if there is no intersection or if the intersection is Returns +inf if there is no intersection or if the intersection is
behind the origin. behind the origin.
"""
if object_['type'] == 'plane':
return intersect_plane(ray_origin, ray_direction,
object_['position'], object_['normal'])
# object_['type'] == 'sphere':
return intersect_sphere(ray_origin, ray_direction,
object_['position'], object_['radius'])
"""
if object_["type"] == "plane":
return intersect_plane(
ray_origin, ray_direction, object_["position"], object_["normal"]
)
# object_['type'] == 'sphere':
return intersect_sphere(
ray_origin, ray_direction, object_["position"], object_["radius"]
)
def get_normal(object_, intersection_point): def get_normal(object_, intersection_point):
""" """Calculate the normal at the intersection point on the object.
Calculate the normal at the intersection point on the object.
Parameters: Parameters
----------
obj (dict): A dictionary representing the object with keys obj (dict): A dictionary representing the object with keys
'type' and 'position'. 'type' and 'position'.
intersection_point (numpy.ndarray): A 3D point representing the intersection_point (numpy.ndarray): A 3D point representing the
intersection point on the object. intersection point on the object.
Returns: Returns
-------
numpy.ndarray: The normal vector at the intersection point. numpy.ndarray: The normal vector at the intersection point.
""" """
if object_['type'] == 'sphere': if object_["type"] == "sphere":
normal = normalize(intersection_point - object_['position']) normal = normalize(intersection_point - object_["position"])
elif object_['type'] == 'plane': elif object_["type"] == "plane":
normal = object_['normal'] normal = object_["normal"]
else: else:
raise ValueError(f"Unknown object type: {object_['type']}") raise ValueError(f"Unknown object type: {object_['type']}")
return normal return normal
def get_color(object_, intersection_point): def get_color(object_, intersection_point):
""" """Get the color of the object at the intersection point.
Get the color of the object at the intersection point.
Parameters: Parameters
----------
object_ (dict): A dictionary representing the object with a key 'color'. object_ (dict): A dictionary representing the object with a key 'color'.
intersection_point (numpy.ndarray): A 3D point representing the intersection_point (numpy.ndarray): A 3D point representing the
intersection point on the object. intersection point on the object.
Returns: Returns
-------
numpy.ndarray: The color of the object at the intersection point. numpy.ndarray: The color of the object at the intersection point.
""" """
color = object_['color'] color = object_["color"]
if not hasattr(color, '__len__'): if not hasattr(color, "__len__"):
color = color(intersection_point) color = color(intersection_point)
return color return color
def trace_ray(ray_origin, ray_direction): def trace_ray(ray_origin, ray_direction):
""" """Trace a ray and find the color at the intersection point.
Trace a ray and find the color at the intersection point.
Parameters: Parameters
----------
ray_origin (numpy.ndarray): A 3D point representing the origin of the ray. ray_origin (numpy.ndarray): A 3D point representing the origin of the ray.
ray_direction (numpy.ndarray): ray_direction (numpy.ndarray):
A normalized 3D vector representing the direction of the ray. A normalized 3D vector representing the direction of the ray.
Returns: Returns
-------
tuple: A tuple containing the object, tuple: A tuple containing the object,
intersection point, normal at the intersection, intersection point, normal at the intersection,
and the color at the intersection point. and the color at the intersection point.
Returns None if there is no intersection. Returns None if there is no intersection.
""" """
t, obj_idx = find_intersection(ray_origin, ray_direction) t, obj_idx = find_intersection(ray_origin, ray_direction)
if t == np.inf: if t == np.inf:
return None return None
object_, intersection_point = get_intersection_details( object_, intersection_point = get_intersection_details(
ray_origin, ray_direction, t, obj_idx) ray_origin, ray_direction, t, obj_idx
normal, color = get_normal(object_, intersection_point), get_color( )
object_, intersection_point) normal, color = (
get_normal(object_, intersection_point),
get_color(object_, intersection_point),
)
if is_shadowed(intersection_point, normal, obj_idx): if is_shadowed(intersection_point, normal, obj_idx):
return None return None
return compute_color( return compute_color(object_, intersection_point, normal, color, ray_origin)
object_, intersection_point, normal, color, ray_origin)
def find_intersection(ray_origin, ray_direction): def find_intersection(ray_origin, ray_direction):
""" """Find the intersection of a ray with the objects in the scene.
Find the intersection of a ray with the objects in the scene.
Parameters: Parameters
----------
ray_origin (numpy.ndarray): A 3D point representing the origin of the ray. ray_origin (numpy.ndarray): A 3D point representing the origin of the ray.
ray_direction (numpy.ndarray): ray_direction (numpy.ndarray):
A normalized 3D vector representing the direction of the ray. A normalized 3D vector representing the direction of the ray.
Returns: Returns
-------
tuple: A tuple containing the distance to the intersection point tuple: A tuple containing the distance to the intersection point
and the index of the intersected object. and the index of the intersected object.
""" """
t = np.inf t = np.inf
obj_idx = -1 obj_idx = -1
@ -223,52 +241,55 @@ def ray_trace(num_spheres, environment, image_width=400, image_height=300, outpu
t, obj_idx = t_obj, index t, obj_idx = t_obj, index
return t, obj_idx return t, obj_idx
def get_intersection_details(ray_origin, ray_direction, t, obj_idx): def get_intersection_details(ray_origin, ray_direction, t, obj_idx):
""" """Get the details of the intersection point on the object.
Get the details of the intersection point on the object.
Parameters: Parameters
----------
ray_origin (numpy.ndarray): A 3D point representing the origin of the ray. ray_origin (numpy.ndarray): A 3D point representing the origin of the ray.
ray_direction (numpy.ndarray): ray_direction (numpy.ndarray):
A normalized 3D vector representing the direction of the ray. A normalized 3D vector representing the direction of the ray.
t (float): The distance to the intersection point. t (float): The distance to the intersection point.
obj_idx (int): The index of the intersected object in the scene. obj_idx (int): The index of the intersected object in the scene.
Returns: Returns
-------
tuple: A tuple containing the intersected object tuple: A tuple containing the intersected object
and the intersection point. and the intersection point.
""" """
object_ = scene[obj_idx] object_ = scene[obj_idx]
intersection_point = ray_origin + ray_direction * t intersection_point = ray_origin + ray_direction * t
return object_, intersection_point return object_, intersection_point
def is_shadowed(intersection_point, normal, obj_idx): def is_shadowed(intersection_point, normal, obj_idx):
""" """Determine if the intersection point is in shadow.
Determine if the intersection point is in shadow.
Parameters: Parameters
----------
intersection_point (numpy.ndarray): intersection_point (numpy.ndarray):
A 3D point representing the intersection point on the object. A 3D point representing the intersection point on the object.
normal (numpy.ndarray): The normal vector at the intersection point. normal (numpy.ndarray): The normal vector at the intersection point.
obj_idx (int): The index of the intersected object in the scene. obj_idx (int): The index of the intersected object in the scene.
Returns: Returns
-------
bool: True if the intersection point is in shadow, False otherwise. bool: True if the intersection point is in shadow, False otherwise.
""" """
to_light = normalize(L - intersection_point) to_light = normalize(L - intersection_point)
shadow_intersections = [intersect( shadow_intersections = [
intersection_point + normal * .0001, to_light, obj_sh) intersect(intersection_point + normal * 0.0001, to_light, obj_sh)
for k, obj_sh in enumerate(scene) if k != obj_idx] for k, obj_sh in enumerate(scene)
if k != obj_idx
]
return shadow_intersections and min(shadow_intersections) < np.inf return shadow_intersections and min(shadow_intersections) < np.inf
def compute_color(object_, intersection_point, normal, color, ray_origin): def compute_color(object_, intersection_point, normal, color, ray_origin):
""" """Compute the color at the intersection point using shading techniques.
Compute the color at the intersection point using shading techniques.
Parameters: Parameters
----------
object_ (dict): A dictionary representing the intersected object. object_ (dict): A dictionary representing the intersected object.
intersection_point (numpy.ndarray): intersection_point (numpy.ndarray):
A 3D point representing the intersection point on the object. A 3D point representing the intersection point on the object.
@ -276,85 +297,94 @@ def ray_trace(num_spheres, environment, image_width=400, image_height=300, outpu
color (numpy.ndarray): The base color of the object. color (numpy.ndarray): The base color of the object.
ray_origin (numpy.ndarray): A 3D point representing the origin of the ray. ray_origin (numpy.ndarray): A 3D point representing the origin of the ray.
Returns: Returns
-------
tuple: tuple:
A tuple containing the intersected object, intersection point, normal, A tuple containing the intersected object, intersection point, normal,
and the computed color. and the computed color.
""" """
to_light = normalize(L - intersection_point) to_light = normalize(L - intersection_point)
to_origin = normalize(ray_origin - intersection_point) to_origin = normalize(ray_origin - intersection_point)
color_ray = AMBIENT color_ray = AMBIENT
diffuse_intensity = object_.get('diffuse_c', DIFFUSE_C) * max( diffuse_intensity = object_.get("diffuse_c", DIFFUSE_C) * max(
np.dot(normal, to_light), 0) np.dot(normal, to_light), 0
)
color_ray += diffuse_intensity * color color_ray += diffuse_intensity * color
half_vector = normalize(to_light + to_origin) half_vector = normalize(to_light + to_origin)
specular_intensity = object_.get('specular_c', SPECULAR_C) * max( specular_intensity = (
np.dot(normal, half_vector), 0) ** SPECULAR_K object_.get("specular_c", SPECULAR_C)
* max(np.dot(normal, half_vector), 0) ** SPECULAR_K
)
color_ray += specular_intensity * color_light color_ray += specular_intensity * color_light
return object_, intersection_point, normal, color_ray return object_, intersection_point, normal, color_ray
def add_sphere(position, radius, color): def add_sphere(position, radius, color):
""" """Create a dictionary representing a sphere object.
Create a dictionary representing a sphere object.
Parameters: Parameters
----------
position (list or numpy.ndarray): position (list or numpy.ndarray):
A 3D point representing the position of the sphere. A 3D point representing the position of the sphere.
radius (float): The radius of the sphere. radius (float): The radius of the sphere.
color (list or numpy.ndarray): The color of the sphere. color (list or numpy.ndarray): The color of the sphere.
Returns: Returns
-------
dict: A dictionary representing the sphere object. dict: A dictionary representing the sphere object.
""" """
return { return {
'type': 'sphere', "type": "sphere",
'position': np.array(position), "position": np.array(position),
'radius': np.array(radius), "radius": np.array(radius),
'color': np.array(color), "color": np.array(color),
'reflection': .5 "reflection": 0.5,
} }
def add_plane(position, normal): def add_plane(position, normal):
""" """Create a dictionary representing a plane object.
Create a dictionary representing a plane object.
Parameters: Parameters
----------
position (list or numpy.ndarray): position (list or numpy.ndarray):
A 3D point representing a point on the plane. A 3D point representing a point on the plane.
normal (list or numpy.ndarray): normal (list or numpy.ndarray):
A normalized 3D vector representing the normal of the plane. A normalized 3D vector representing the normal of the plane.
Returns: Returns
-------
dict: A dictionary representing the plane object. dict: A dictionary representing the plane object.
""" """
return { return {
'type': 'plane', "type": "plane",
'position': np.array(position), "position": np.array(position),
'normal': np.array(normal), "normal": np.array(normal),
'color': lambda M: (color_plane0 "color": lambda M: (
if (int(M[0] * 2) % 2) == (int(M[2] * 2) % 2) color_plane0
else color_plane1), if (int(M[0] * 2) % 2) == (int(M[2] * 2) % 2)
'diffuse_c': .75, else color_plane1
'specular_c': .5, ),
'reflection': .25 "diffuse_c": 0.75,
"specular_c": 0.5,
"reflection": 0.25,
} }
scene = [] scene = []
# List of objects. # List of objects.
color_plane0 = 1. * np.ones(3) color_plane0 = 1.0 * np.ones(3)
color_plane1 = 0. * np.ones(3) color_plane1 = 0.0 * np.ones(3)
scene.append(add_plane([0., -0.5, 0.], [0., 1., 0.])) scene.append(add_plane([0.0, -0.5, 0.0], [0.0, 1.0, 0.0]))
base_radius = 1 / np.sqrt(num_spheres) # Im więcej kul, tym mniejsze base_radius = 1 / np.sqrt(num_spheres) # Im więcej kul, tym mniejsze
base_distance = 4.5 / num_spheres base_distance = 4.5 / num_spheres
for i in range(num_spheres): for i in range(num_spheres):
# Wyliczanie pozycji każdej kuli # Wyliczanie pozycji każdej kuli
x = (i - num_spheres // 2) * base_distance x = (i - num_spheres // 2) * base_distance
y = 0.1 y = 0.1
z = 1. + i * 0.5 z = 1.0 + i * 0.5
# Dynamiczny kolor (gradient na podstawie indeksu) # Dynamiczny kolor (gradient na podstawie indeksu)
color = np.array([i / num_spheres, (num_spheres - i) / num_spheres, 0.5]) color = np.array([i / num_spheres, (num_spheres - i) / num_spheres, 0.5])
@ -363,24 +393,24 @@ def ray_trace(num_spheres, environment, image_width=400, image_height=300, outpu
scene.append(add_sphere([x, y, z], base_radius, color)) scene.append(add_sphere([x, y, z], base_radius, color))
# Light position and color. # Light position and color.
L = np.array([5., 5., -10.]) L = np.array([5.0, 5.0, -10.0])
color_light = np.ones(3) color_light = np.ones(3)
# Default light and material parameters. # Default light and material parameters.
AMBIENT = .05 AMBIENT = 0.05
DIFFUSE_C = 1. DIFFUSE_C = 1.0
SPECULAR_C = 1. SPECULAR_C = 1.0
SPECULAR_K = 50 SPECULAR_K = 50
DEPTH_MAX = 5 # Maximum number of light reflections. DEPTH_MAX = 5 # Maximum number of light reflections.
col = np.zeros(3) # Current color. col = np.zeros(3) # Current color.
camera_origin = np.array([0., 0.35, -1.]) # Camera. camera_origin = np.array([0.0, 0.35, -1.0]) # Camera.
Q = np.array([0., 0., 0.]) # Camera pointing to. Q = np.array([0.0, 0.0, 0.0]) # Camera pointing to.
img = np.zeros((IMAGE_HEIGHT, IMAGE_WIDTH, 3)) img = np.zeros((IMAGE_HEIGHT, IMAGE_WIDTH, 3))
r = float(IMAGE_WIDTH) / IMAGE_HEIGHT r = float(IMAGE_WIDTH) / IMAGE_HEIGHT
# Screen coordinates: x0, y0, x1, y1. # Screen coordinates: x0, y0, x1, y1.
S = (-1., -1. / r + .25, 1., 1. / r + .25) S = (-1.0, -1.0 / r + 0.25, 1.0, 1.0 / r + 0.25)
renderTime = time.time() renderTime = time.time()
reflections = 0 reflections = 0
@ -396,7 +426,7 @@ def ray_trace(num_spheres, environment, image_width=400, image_height=300, outpu
D = normalize(Q - camera_origin) D = normalize(Q - camera_origin)
DEPTH = 0 DEPTH = 0
rayO, rayD = camera_origin, D rayO, rayD = camera_origin, D
REFLECTION = 1. REFLECTION = 1.0
initialRays += 1 initialRays += 1
# Loop through initial and secondary rays. # Loop through initial and secondary rays.
while DEPTH < DEPTH_MAX: while DEPTH < DEPTH_MAX:
@ -407,19 +437,20 @@ def ray_trace(num_spheres, environment, image_width=400, image_height=300, outpu
reflections += 1 reflections += 1
obj, M, N, col_ray = traced obj, M, N, col_ray = traced
# Reflection: create a new ray. # Reflection: create a new ray.
rayO, rayD = M + \ rayO, rayD = M + N * 0.0001, normalize(rayD - 2 * np.dot(rayD, N) * N)
N * .0001, normalize(rayD - 2 * np.dot(rayD, N) * N)
DEPTH += 1 DEPTH += 1
col += REFLECTION * col_ray col += REFLECTION * col_ray
REFLECTION *= obj.get('reflection', 1.) REFLECTION *= obj.get("reflection", 1.0)
img[IMAGE_HEIGHT - j - 1, i, :] = np.clip(col, 0, 1) img[IMAGE_HEIGHT - j - 1, i, :] = np.clip(col, 0, 1)
renderTime = time.time() - renderTime renderTime = time.time() - renderTime
plt.imsave(output_file, img) plt.imsave(output_file, img)
print(f"Image saved as {output_file}\n" print(
f"resolution: {IMAGE_WIDTH}x{IMAGE_HEIGHT}\n" f"Image saved as {output_file}\n"
f"render time: {round(renderTime, 2)} s\n" f"resolution: {IMAGE_WIDTH}x{IMAGE_HEIGHT}\n"
f"reflections: {reflections}\n" f"render time: {round(renderTime, 2)} s\n"
f"rays (initial): {initialRays}\n" f"reflections: {reflections}\n"
f"rays (secondary): {rays - initialRays}\n" f"rays (initial): {initialRays}\n"
f"rays (total): {rays}") f"rays (secondary): {rays - initialRays}\n"
f"rays (total): {rays}"
)

11
Programming/TRAK/run.sh
View File

@ -7,5 +7,16 @@ set -euo pipefail
# shellcheck source=../../scripts/course_lib.sh # shellcheck source=../../scripts/course_lib.sh
. "$(dirname "$(readlink -f "$0")")/../../scripts/course_lib.sh" . "$(dirname "$(readlink -f "$0")")/../../scripts/course_lib.sh"
# Three main.py files live here (the renderer, plus the two standalone
# experiments under code/), so the harness cannot pick one on its own and
# falls back to syntax-checking. Name the renderer explicitly: a run that
# actually ray-traces cornell_box at 400x300 is a real check, and "every
# module parses" is not.
# --output, because config.ini's default is outputs/output.png and that file
# is a TRACKED deliverable: a plain run silently overwrote what was submitted.
# build/ is gitignored, so a rebuild can never do that again.
mkdir -p "$(dirname "$(readlink -f "$0")")/build"
course_main --kind python \ course_main --kind python \
--entry "$(dirname "$(readlink -f "$0")")/main.py" \
--entry-args "--output ../build/render.png" \
-- "$@" -- "$@"

1
Programming/TRAK/scenes/cornell_box.py
View File

@ -1,5 +1,6 @@
from sightpy import * from sightpy import *
# define materials to use # define materials to use
def setup_scene(width=400, height=300, environment=None): def setup_scene(width=400, height=300, environment=None):

24
Programming/TRAK/scenes/soap_bubble.py
View File

@ -1,5 +1,6 @@
from sightpy import * from sightpy import *
# define materials to use # define materials to use
def setup_scene(width=400, height=300, environment="lake.png"): def setup_scene(width=400, height=300, environment="lake.png"):
# Set Scene # Set Scene
@ -7,13 +8,24 @@ def setup_scene(width=400, height=300, environment="lake.png"):
Sc = Scene(ambient_color=rgb(0.01, 0.01, 0.01)) Sc = Scene(ambient_color=rgb(0.01, 0.01, 0.01))
angle = -np.pi * 0.5 angle = -np.pi * 0.5
Sc.add_Camera(screen_height=height, screen_width=width, Sc.add_Camera(
look_from=vec3(4.0 * np.sin(angle), 0.00, 4.0 * np.cos(angle)), screen_height=height,
look_at=vec3(0., 0.05, 0.0)) screen_width=width,
look_from=vec3(4.0 * np.sin(angle), 0.00, 4.0 * np.cos(angle)),
look_at=vec3(0.0, 0.05, 0.0),
)
soap_bubble = ThinFilmInterference(thickness=330, noise=60.) soap_bubble = ThinFilmInterference(thickness=330, noise=60.0)
Sc.add(Sphere(material=soap_bubble, center=vec3(1., 0.0, 1.5), radius=1.7, shadow=False, max_ray_depth=5)) Sc.add(
Sphere(
material=soap_bubble,
center=vec3(1.0, 0.0, 1.5),
radius=1.7,
shadow=False,
max_ray_depth=5,
)
)
Sc.add_Background(environment, blur=10.) Sc.add_Background(environment, blur=10.0)
return Sc return Sc

71
Programming/TRAK/scenes/three_spheres.py
View File

@ -1,33 +1,76 @@
from sightpy import * from sightpy import *
# define materials to use # define materials to use
def setup_scene(width=400, height=300, environment="miramar.jpeg"): def setup_scene(width=400, height=300, environment="miramar.jpeg"):
blue_glass = Refractive(n=vec3(1.5 + 4e-8j, 1.5 + 4e-8j, 1.5 + 0.j)) # n = index of refraction blue_glass = Refractive(
green_glass = Refractive(n=vec3(1.5 + 4e-8j, 1.5 + 0.j, 1.5 + 4e-8j)) n=vec3(1.5 + 4e-8j, 1.5 + 4e-8j, 1.5 + 0.0j)
red_glass = Refractive(n=vec3(1.5 + 0.j, 1.5 + 5e-8j, 1.5 + 5e-8j)) ) # n = index of refraction
green_glass = Refractive(n=vec3(1.5 + 4e-8j, 1.5 + 0.0j, 1.5 + 4e-8j))
red_glass = Refractive(n=vec3(1.5 + 0.0j, 1.5 + 5e-8j, 1.5 + 5e-8j))
floor = Glossy(diff_color=image("checkered_floor.png", repeat=80.), n=vec3(1.2 + 0.3j, 1.2 + 0.3j, 1.1 + 0.3j), floor = Glossy(
roughness=0.2, spec_coeff=0.3, diff_coeff=0.9) diff_color=image("checkered_floor.png", repeat=80.0),
n=vec3(1.2 + 0.3j, 1.2 + 0.3j, 1.1 + 0.3j),
roughness=0.2,
spec_coeff=0.3,
diff_coeff=0.9,
)
# Set Scene # Set Scene
Sc = Scene(ambient_color=rgb(0.05, 0.05, 0.05)) Sc = Scene(ambient_color=rgb(0.05, 0.05, 0.05))
angle = np.pi / 2 * 0.3 angle = np.pi / 2 * 0.3
Sc.add_Camera(look_from=vec3(2.5 * np.sin(angle), 0.25, 2.5 * np.cos(angle) - 1.5), Sc.add_Camera(
look_at=vec3(0., 0.25, -1.5), look_from=vec3(2.5 * np.sin(angle), 0.25, 2.5 * np.cos(angle) - 1.5),
screen_width=width, look_at=vec3(0.0, 0.25, -1.5),
screen_height=height) screen_width=width,
screen_height=height,
)
Sc.add_DirectionalLight(Ldir=vec3(0.52, 0.45, -0.5), color=rgb(0.15, 0.15, 0.15)) Sc.add_DirectionalLight(Ldir=vec3(0.52, 0.45, -0.5), color=rgb(0.15, 0.15, 0.15))
Sc.add(Sphere(material=blue_glass, center=vec3(-1.2, 0.0, -1.5), radius=.5, shadow=False, max_ray_depth=3)) Sc.add(
Sc.add(Sphere(material=green_glass, center=vec3(0., 0.0, -1.5), radius=.5, shadow=False, max_ray_depth=3)) Sphere(
Sc.add(Sphere(material=red_glass, center=vec3(1.2, 0.0, -1.5), radius=.5, shadow=False, max_ray_depth=3)) material=blue_glass,
center=vec3(-1.2, 0.0, -1.5),
radius=0.5,
shadow=False,
max_ray_depth=3,
)
)
Sc.add(
Sphere(
material=green_glass,
center=vec3(0.0, 0.0, -1.5),
radius=0.5,
shadow=False,
max_ray_depth=3,
)
)
Sc.add(
Sphere(
material=red_glass,
center=vec3(1.2, 0.0, -1.5),
radius=0.5,
shadow=False,
max_ray_depth=3,
)
)
Sc.add(Plane(material=floor, center=vec3(0, -0.5, -3.0), width=120.0, height=120.0, u_axis=vec3(1.0, 0, 0), Sc.add(
v_axis=vec3(0, 0, -1.0), max_ray_depth=3)) Plane(
material=floor,
center=vec3(0, -0.5, -3.0),
width=120.0,
height=120.0,
u_axis=vec3(1.0, 0, 0),
v_axis=vec3(0, 0, -1.0),
max_ray_depth=3,
)
)
# see sightpy/backgrounds # see sightpy/backgrounds
Sc.add_Background(environment) Sc.add_Background(environment)

45
Programming/TRAK/scenes/transparent_cuboid.py
View File

@ -1,23 +1,48 @@
from sightpy import * from sightpy import *
# define materials to use # define materials to use
def setup_scene(width=400, height=300, environment="stormydays.png"): def setup_scene(width=400, height=300, environment="stormydays.png"):
floor = Glossy(diff_color=image("checkered_floor.png", repeat=2.), roughness=0.2, spec_coeff=0.3, diff_coeff=0.7, floor = Glossy(
n=vec3(2.2, 2.2, 2.2)) # n = index of refraction diff_color=image("checkered_floor.png", repeat=2.0),
green_glass = Refractive(n=vec3(1.5 + 4e-8j, 1.5 + 0.j, 1.5 + 4e-8j)) roughness=0.2,
spec_coeff=0.3,
diff_coeff=0.7,
n=vec3(2.2, 2.2, 2.2),
) # n = index of refraction
green_glass = Refractive(n=vec3(1.5 + 4e-8j, 1.5 + 0.0j, 1.5 + 4e-8j))
Sc = Scene() Sc = Scene()
Sc.add_Camera(look_from=vec3(0., 0.25, 1.), look_at=vec3(0., 0.25, -3.), Sc.add_Camera(
screen_width=width, look_from=vec3(0.0, 0.25, 1.0),
screen_height=height) look_at=vec3(0.0, 0.25, -3.0),
screen_width=width,
screen_height=height,
)
Sc.add_DirectionalLight(Ldir=vec3(0.0, 0.5, 0.5), color=rgb(0.5, 0.5, 0.5)) Sc.add_DirectionalLight(Ldir=vec3(0.0, 0.5, 0.5), color=rgb(0.5, 0.5, 0.5))
Sc.add(Plane(material=floor, center=vec3(0, -0.5, -3.0), width=6.0, height=6.0, u_axis=vec3(1.0, 0, 0), Sc.add(
v_axis=vec3(0, 0, -1.0), max_ray_depth=5)) Plane(
material=floor,
center=vec3(0, -0.5, -3.0),
width=6.0,
height=6.0,
u_axis=vec3(1.0, 0, 0),
v_axis=vec3(0, 0, -1.0),
max_ray_depth=5,
)
)
cb = Cuboid(material=green_glass, center=vec3(0.00, 0.0001, -0.8), width=0.9, height=1.0, length=0.4, shadow=False, cb = Cuboid(
max_ray_depth=5) material=green_glass,
center=vec3(0.00, 0.0001, -0.8),
width=0.9,
height=1.0,
length=0.4,
shadow=False,
max_ray_depth=5,
)
cb.rotate(θ=30, u=vec3(0, 1, 0)) cb.rotate(θ=30, u=vec3(0, 1, 0))
Sc.add(cb) Sc.add(cb)

82
Programming/TRAK/scenes/two_spheres.py
View File

@ -1,38 +1,74 @@
from sightpy import * from sightpy import *
# define materials to use # define materials to use
def setup_scene(width=400, height=300, environment="stormydays.png"): def setup_scene(width=400, height=300, environment="stormydays.png"):
gold_metal = Glossy(diff_color = rgb(1., .572, .184), n = vec3(0.15+3.58j, 0.4+2.37j, 1.54+1.91j), roughness = 0.0, spec_coeff = 0.2, diff_coeff= 0.8) # n = index of refraction gold_metal = Glossy(
bluish_metal = Glossy(diff_color = rgb(0.0, 0, 0.1), n = vec3(1.3+1.91j, 1.3+1.91j, 1.4+2.91j), roughness = 0.2,spec_coeff = 0.5, diff_coeff= 0.3) diff_color=rgb(1.0, 0.572, 0.184),
n=vec3(0.15 + 3.58j, 0.4 + 2.37j, 1.54 + 1.91j),
floor = Glossy(diff_color = image("checkered_floor.png", repeat = 80.), roughness=0.0,
n = vec3(1.2+ 0.3j, 1.2+ 0.3j, 1.1+ 0.3j), roughness = 0.2, spec_coeff = 0.3, diff_coeff= 0.9 ) spec_coeff=0.2,
diff_coeff=0.8,
) # n = index of refraction
bluish_metal = Glossy(
diff_color=rgb(0.0, 0, 0.1),
n=vec3(1.3 + 1.91j, 1.3 + 1.91j, 1.4 + 2.91j),
roughness=0.2,
spec_coeff=0.5,
diff_coeff=0.3,
)
floor = Glossy(
diff_color=image("checkered_floor.png", repeat=80.0),
n=vec3(1.2 + 0.3j, 1.2 + 0.3j, 1.1 + 0.3j),
roughness=0.2,
spec_coeff=0.3,
diff_coeff=0.9,
)
# Set Scene # Set Scene
Sc = Scene(ambient_color = rgb(0.05, 0.05, 0.05)) Sc = Scene(ambient_color=rgb(0.05, 0.05, 0.05))
angle = -np.pi / 2 * 0.3
Sc.add_Camera(
look_from=vec3(2.5 * np.sin(angle), 0.25, 2.5 * np.cos(angle) - 1.5),
look_at=vec3(0.0, 0.25, -3.0),
screen_width=width,
screen_height=height,
)
angle = -np.pi/2 * 0.3 Sc.add_DirectionalLight(Ldir=vec3(0.52, 0.45, -0.5), color=rgb(0.15, 0.15, 0.15))
Sc.add_Camera(look_from = vec3(2.5*np.sin(angle), 0.25, 2.5*np.cos(angle) -1.5 ),
look_at = vec3(0., 0.25, -3.),
screen_width = width ,
screen_height = height)
Sc.add(
Sphere(
material=gold_metal,
center=vec3(-0.75, 0.1, -3.0),
radius=0.6,
max_ray_depth=3,
)
)
Sc.add(
Sphere(
material=bluish_metal,
center=vec3(1.25, 0.1, -3.0),
radius=0.6,
max_ray_depth=3,
)
)
Sc.add(
Plane(
material=floor,
center=vec3(0, -0.5, -3.0),
width=120.0,
height=120.0,
u_axis=vec3(1.0, 0, 0),
v_axis=vec3(0, 0, -1.0),
max_ray_depth=3,
)
)
Sc.add_DirectionalLight(Ldir = vec3(0.52,0.45, -0.5), color = rgb(0.15, 0.15, 0.15)) # see sightpy/backgrounds
Sc.add(Sphere(material = gold_metal, center = vec3(-.75, .1, -3.),radius = .6, max_ray_depth = 3))
Sc.add(Sphere(material = bluish_metal, center = vec3(1.25, .1, -3.), radius = .6, max_ray_depth = 3))
Sc.add(Plane(material = floor, center = vec3(0, -0.5, -3.0), width = 120.0,height = 120.0, u_axis = vec3(1.0, 0, 0), v_axis = vec3(0, 0, -1.0), max_ray_depth = 3))
#see sightpy/backgrounds
Sc.add_Background(environment) Sc.add_Background(environment)
return Sc return Sc

8
Programming/TRAK/utils.py
View File

@ -1,13 +1,17 @@
from configparser import ConfigParser from configparser import ConfigParser
def load_config(config_path): def load_config(config_path):
config = ConfigParser() config = ConfigParser()
config.read(config_path) config.read(config_path)
return config return config
def parse_resolution(resolution): def parse_resolution(resolution):
try: try:
width, height = map(int, resolution.lower().split('x')) width, height = map(int, resolution.lower().split("x"))
return width, height return width, height
except ValueError: except ValueError:
raise ValueError("Resolution must be in the format WIDTHxHEIGHT, e.g., 1920x1080.") raise ValueError(
"Resolution must be in the format WIDTHxHEIGHT, e.g., 1920x1080."
)

21
Programming/WDWR/projekt/data/pareto_front.csv Normal file
View File

@ -0,0 +1,21 @@
i;minimalAverageProfit;averageProfit;riskMeasureGini;m1_prod_P1;m1_prod_P2;m1_prod_P3;m1_prod_P4;m2_prod_P1;m2_prod_P2;m2_prod_P3;m2_prod_P4;m3_prod_P1;m3_prod_P2;m3_prod_P3;m3_prod_P4;m1_stock_P1;m1_stock_P2;m1_stock_P3;m1_stock_P4;m2_stock_P1;m2_stock_P2;m2_stock_P3;m2_stock_P4;m3_stock_P1;m3_stock_P2;m3_stock_P3;m3_stock_P4
1;-600;-600;0;0;0;0;0;0;0;0;0;0;0;0;0;50;50;50;50;50;50;50;50;50;50;50;50
2;-67.59;-67.423334025;12.249866065;0;0;0;1;0;0;0;0;9;8;1;43;41;50;50;7;41;42;49;7;50;50;50;50
3;464.82;465.647226284;26.015138747;0;0;0;0;0;0;0;29;20;19;9;52;30;50;41;0;30;31;41;0;50;50;50;50
4;997.23;997.66185313;40.349353614;0;0;0;110;0;0;0;0;30;30;13;50;20;50;37;0;20;20;37;0;50;50;50;50
5;1529.64;1530.172175866;54.689711726;0;0;0;114;0;0;0;0;42;40;17;92;8;50;33;0;8;10;33;0;50;50;50;50
6;2062.05;2062.088549477;69.062913221;0;0;0;9;0;0;0;200;50;50;25;50;0;50;25;0;0;0;25;0;50;50;50;50
7;2594.46;2594.684714046;84.265583873;9;0;0;64;0;10;0;159;50;51;28;51;0;50;22;0;0;0;22;0;50;50;50;50
8;3126.87;3127.12427492;99.497458717;20;0;0;0;0;30;0;111;50;58;35;249;0;50;15;0;0;0;15;0;50;50;50;50
9;3659.28;3659.345897427;114.728718529;29;0;0;2;1;31;0;200;50;69;39;249;0;50;11;0;0;0;11;0;50;50;50;50
10;4191.69;4191.916591732;129.963670504;36;0;0;114;5;32;0;160;50;79;45;210;0;50;5;0;0;0;5;0;50;50;50;50
11;4724.1;4724.191045823;145.204545226;50;0;0;100;2;34;0;197;50;89;50;250;0;50;0;0;0;0;0;0;50;50;50;50
12;5256.51;5256.719240584;162.349562672;72;0;34;150;0;3;0;200;50;123;50;210;0;50;0;0;0;0;0;0;50;50;50;50
13;5788.92;5789.016098305;182.46075748;122;0;0;150;1;1;0;200;50;149;94;210;0;50;0;0;0;0;0;0;50;50;50;50
14;6321.33;6321.358645932;205.000585625;110;0;50;150;19;33;0;200;50;156;102;250;0;50;0;0;0;0;0;0;50;50;50;50
15;6853.74;6853.83219788;229.023498807;109;0;24;110;1;0;94;160;50;191;51;250;0;50;0;0;0;0;0;0;50;50;50;50
16;7386.15;7386.171785273;253.962965038;1;0;1;150;121;30;189;200;50;200;50;210;0;50;0;0;0;0;0;0;50;50;50;50
17;7918.56;7918.565798346;279.907638363;110;0;30;110;59;48;76;160;50;205;130;210;0;50;0;0;0;0;0;0;50;50;50;50
18;8450.97;8451.020747071;306.383357869;111;0;0;150;72;0;191;200;50;256;131;210;0;50;0;0;0;0;0;0;50;50;50;50
19;8983.38;8983.400103901;332.936696297;111;0;49;110;83;15;144;200;50;317;149;250;0;50;0;0;0;0;0;0;50;50;50;50
20;9515.79;9515.804890325;360.18213126;110;0;30;110;110;28;160;160;50;273;150;250;0;50;0;0;0;0;0;0;50;50;50;50
1 i minimalAverageProfit averageProfit riskMeasureGini m1_prod_P1 m1_prod_P2 m1_prod_P3 m1_prod_P4 m2_prod_P1 m2_prod_P2 m2_prod_P3 m2_prod_P4 m3_prod_P1 m3_prod_P2 m3_prod_P3 m3_prod_P4 m1_stock_P1 m1_stock_P2 m1_stock_P3 m1_stock_P4 m2_stock_P1 m2_stock_P2 m2_stock_P3 m2_stock_P4 m3_stock_P1 m3_stock_P2 m3_stock_P3 m3_stock_P4
2 1 -600 -600 0 0 0 0 0 0 0 0 0 0 0 0 0 50 50 50 50 50 50 50 50 50 50 50 50
3 2 -67.59 -67.423334025 12.249866065 0 0 0 1 0 0 0 0 9 8 1 43 41 50 50 7 41 42 49 7 50 50 50 50
4 3 464.82 465.647226284 26.015138747 0 0 0 0 0 0 0 29 20 19 9 52 30 50 41 0 30 31 41 0 50 50 50 50
5 4 997.23 997.66185313 40.349353614 0 0 0 110 0 0 0 0 30 30 13 50 20 50 37 0 20 20 37 0 50 50 50 50
6 5 1529.64 1530.172175866 54.689711726 0 0 0 114 0 0 0 0 42 40 17 92 8 50 33 0 8 10 33 0 50 50 50 50
7 6 2062.05 2062.088549477 69.062913221 0 0 0 9 0 0 0 200 50 50 25 50 0 50 25 0 0 0 25 0 50 50 50 50
8 7 2594.46 2594.684714046 84.265583873 9 0 0 64 0 10 0 159 50 51 28 51 0 50 22 0 0 0 22 0 50 50 50 50
9 8 3126.87 3127.12427492 99.497458717 20 0 0 0 0 30 0 111 50 58 35 249 0 50 15 0 0 0 15 0 50 50 50 50
10 9 3659.28 3659.345897427 114.728718529 29 0 0 2 1 31 0 200 50 69 39 249 0 50 11 0 0 0 11 0 50 50 50 50
11 10 4191.69 4191.916591732 129.963670504 36 0 0 114 5 32 0 160 50 79 45 210 0 50 5 0 0 0 5 0 50 50 50 50
12 11 4724.1 4724.191045823 145.204545226 50 0 0 100 2 34 0 197 50 89 50 250 0 50 0 0 0 0 0 0 50 50 50 50
13 12 5256.51 5256.719240584 162.349562672 72 0 34 150 0 3 0 200 50 123 50 210 0 50 0 0 0 0 0 0 50 50 50 50
14 13 5788.92 5789.016098305 182.46075748 122 0 0 150 1 1 0 200 50 149 94 210 0 50 0 0 0 0 0 0 50 50 50 50
15 14 6321.33 6321.358645932 205.000585625 110 0 50 150 19 33 0 200 50 156 102 250 0 50 0 0 0 0 0 0 50 50 50 50
16 15 6853.74 6853.83219788 229.023498807 109 0 24 110 1 0 94 160 50 191 51 250 0 50 0 0 0 0 0 0 50 50 50 50
17 16 7386.15 7386.171785273 253.962965038 1 0 1 150 121 30 189 200 50 200 50 210 0 50 0 0 0 0 0 0 50 50 50 50
18 17 7918.56 7918.565798346 279.907638363 110 0 30 110 59 48 76 160 50 205 130 210 0 50 0 0 0 0 0 0 50 50 50 50
19 18 8450.97 8451.020747071 306.383357869 111 0 0 150 72 0 191 200 50 256 131 210 0 50 0 0 0 0 0 0 50 50 50 50
20 19 8983.38 8983.400103901 332.936696297 111 0 49 110 83 15 144 200 50 317 149 250 0 50 0 0 0 0 0 0 50 50 50 50
21 20 9515.79 9515.804890325 360.18213126 110 0 30 110 110 28 160 160 50 273 150 250 0 50 0 0 0 0 0 0 50 50 50 50

92
Programming/WDWR/projekt/graph.py Normal file → Executable file
View File

@ -1,53 +1,59 @@
#!/usr/bin/env python3
"""Plot the Pareto front of the WDWR production model: profit against risk.
Each row of ``data/pareto_front.csv`` is one solution of the bi-criteria MILP
-- an average profit, its Gini risk measure, and the production and stock
levels that produced them. The scatter shows the trade-off between the two
criteria.
The data used to be a 20-line CSV embedded in this file as a string literal.
It moved to a sibling file because its header line alone is 342 characters,
and a data record is not something that can be wrapped to fit a line limit.
"""
import logging
from pathlib import Path
import matplotlib.pyplot as plt import matplotlib.pyplot as plt
import pandas as pd import pandas as pd
from io import StringIO
# Your data logger = logging.getLogger(__name__)
data_str = """i;minimalAverageProfit;averageProfit;riskMeasureGini;m1_prod_P1;m1_prod_P2;m1_prod_P3;m1_prod_P4;m2_prod_P1;m2_prod_P2;m2_prod_P3;m2_prod_P4;m3_prod_P1;m3_prod_P2;m3_prod_P3;m3_prod_P4;m1_stock_P1;m1_stock_P2;m1_stock_P3;m1_stock_P4;m2_stock_P1;m2_stock_P2;m2_stock_P3;m2_stock_P4;m3_stock_P1;m3_stock_P2;m3_stock_P3;m3_stock_P4
1;-600;-600;0;0;0;0;0;0;0;0;0;0;0;0;0;50;50;50;50;50;50;50;50;50;50;50;50
2;-67.59;-67.423334025;12.249866065;0;0;0;1;0;0;0;0;9;8;1;43;41;50;50;7;41;42;49;7;50;50;50;50
3;464.82;465.647226284;26.015138747;0;0;0;0;0;0;0;29;20;19;9;52;30;50;41;0;30;31;41;0;50;50;50;50
4;997.23;997.66185313;40.349353614;0;0;0;110;0;0;0;0;30;30;13;50;20;50;37;0;20;20;37;0;50;50;50;50
5;1529.64;1530.172175866;54.689711726;0;0;0;114;0;0;0;0;42;40;17;92;8;50;33;0;8;10;33;0;50;50;50;50
6;2062.05;2062.088549477;69.062913221;0;0;0;9;0;0;0;200;50;50;25;50;0;50;25;0;0;0;25;0;50;50;50;50
7;2594.46;2594.684714046;84.265583873;9;0;0;64;0;10;0;159;50;51;28;51;0;50;22;0;0;0;22;0;50;50;50;50
8;3126.87;3127.12427492;99.497458717;20;0;0;0;0;30;0;111;50;58;35;249;0;50;15;0;0;0;15;0;50;50;50;50
9;3659.28;3659.345897427;114.728718529;29;0;0;2;1;31;0;200;50;69;39;249;0;50;11;0;0;0;11;0;50;50;50;50
10;4191.69;4191.916591732;129.963670504;36;0;0;114;5;32;0;160;50;79;45;210;0;50;5;0;0;0;5;0;50;50;50;50
11;4724.1;4724.191045823;145.204545226;50;0;0;100;2;34;0;197;50;89;50;250;0;50;0;0;0;0;0;0;50;50;50;50
12;5256.51;5256.719240584;162.349562672;72;0;34;150;0;3;0;200;50;123;50;210;0;50;0;0;0;0;0;0;50;50;50;50
13;5788.92;5789.016098305;182.46075748;122;0;0;150;1;1;0;200;50;149;94;210;0;50;0;0;0;0;0;0;50;50;50;50
14;6321.33;6321.358645932;205.000585625;110;0;50;150;19;33;0;200;50;156;102;250;0;50;0;0;0;0;0;0;50;50;50;50
15;6853.74;6853.83219788;229.023498807;109;0;24;110;1;0;94;160;50;191;51;250;0;50;0;0;0;0;0;0;50;50;50;50
16;7386.15;7386.171785273;253.962965038;1;0;1;150;121;30;189;200;50;200;50;210;0;50;0;0;0;0;0;0;50;50;50;50
17;7918.56;7918.565798346;279.907638363;110;0;30;110;59;48;76;160;50;205;130;210;0;50;0;0;0;0;0;0;50;50;50;50
18;8450.97;8451.020747071;306.383357869;111;0;0;150;72;0;191;200;50;256;131;210;0;50;0;0;0;0;0;0;50;50;50;50
19;8983.38;8983.400103901;332.936696297;111;0;49;110;83;15;144;200;50;317;149;250;0;50;0;0;0;0;0;0;50;50;50;50
20;9515.79;9515.804890325;360.18213126;110;0;30;110;110;28;160;160;50;273;150;250;0;50;0;0;0;0;0;0;50;50;50;50"""
# Read the data DATA_FILE = Path(__file__).parent / "data" / "pareto_front.csv"
df = pd.read_csv(StringIO(data_str), sep=';')
# Create the scatter plot # Everything is sized for a report figure rather than for the screen.
plt.figure(figsize=(10, 6)) LABEL_FONT_SIZE = 16
plt.scatter(df['averageProfit'], df['riskMeasureGini'], alpha=0.7, s=200) MARKER_AREA = 200
# Add labels and title
plt.xlabel('Przeciętny Zysk', fontsize=16)
plt.ylabel('Ryzyko', fontsize=16)
plt.xticks(fontsize=16)
plt.yticks(fontsize=16)
# Add grid for better readability def main() -> None:
plt.grid(True, alpha=0.3) """Draw the scatter plot and report the range of each axis."""
df = pd.read_csv(DATA_FILE, sep=";")
# Format the plot plt.figure(figsize=(10, 6))
plt.tight_layout() plt.scatter(df["averageProfit"], df["riskMeasureGini"], alpha=0.7, s=MARKER_AREA)
# Show the plot plt.xlabel("Przeciętny Zysk", fontsize=LABEL_FONT_SIZE)
plt.show() plt.ylabel("Ryzyko", fontsize=LABEL_FONT_SIZE)
plt.xticks(fontsize=LABEL_FONT_SIZE)
plt.yticks(fontsize=LABEL_FONT_SIZE)
plt.grid(visible=True, alpha=0.3)
plt.tight_layout()
plt.show()
# Print some basic statistics logger.info(
print(f"Average Profit range: {df['averageProfit'].min():.2f} to {df['averageProfit'].max():.2f}") "Average Profit range: %.2f to %.2f",
print(f"Gini Risk range: {df['riskMeasureGini'].min():.2f} to {df['riskMeasureGini'].max():.2f}") df["averageProfit"].min(),
print(f"Number of data points: {len(df)}") df["averageProfit"].max(),
)
logger.info(
"Gini Risk range: %.2f to %.2f",
df["riskMeasureGini"].min(),
df["riskMeasureGini"].max(),
)
logger.info("Number of data points: %d", len(df))
if __name__ == "__main__":
logging.basicConfig(format="%(message)s", level=logging.INFO)
main()

162
Programming/WDWR/projekt/graph2.py Normal file → Executable file
View File

@ -1,32 +1,120 @@
#!/usr/bin/env python3
"""Plot the inverse distribution of profit for the three WDWR scenarios.
One curve per scenario: the sorted profits of its 100 sampled realisations
against their empirical cumulative probability, which is how the report shows
which scenario stochastically dominates the others.
"""
import logging
import matplotlib.pyplot as plt import matplotlib.pyplot as plt
import numpy as np import numpy as np
logger = logging.getLogger(__name__)
logging.basicConfig(format="%(message)s", level=logging.INFO)
# Sized for a report figure rather than for the screen.
LABEL_FONT_SIZE = 16
# Data for the three scenarios # Data for the three scenarios
scenario1_profits = [ scenario1_profits = [
8283.101223531, 8360.092861464, 8023.436774628, 7996.435353438, 8108.734620296, 8283.101223531,
8709.702668322, 9236.429507678, 8492.980846575, 8912.962073143, 8201.079841867, 8360.092861464,
7947.08574225, 8426.386242154, 8633.510288097, 7925.419317516, 8216.653093455, 8023.436774628,
8162.239336848, 10606.276724793, 7766.753696884, 8902.966530729, 8329.954853003, 7996.435353438,
8615.534725088, 9307.905933844, 8711.836200338, 7853.713101162, 8160.850707138, 8108.734620296,
8505.74747647, 7807.599624626, 8650.006339759, 7571.333855925, 9103.892851116 8709.702668322,
9236.429507678,
8492.980846575,
8912.962073143,
8201.079841867,
7947.08574225,
8426.386242154,
8633.510288097,
7925.419317516,
8216.653093455,
8162.239336848,
10606.276724793,
7766.753696884,
8902.966530729,
8329.954853003,
8615.534725088,
9307.905933844,
8711.836200338,
7853.713101162,
8160.850707138,
8505.74747647,
7807.599624626,
8650.006339759,
7571.333855925,
9103.892851116,
] ]
scenario2_profits = [ scenario2_profits = [
8779.880622914, 8905.098118171, 8457.038873118, 8476.337700492, 8611.662470696, 8779.880622914,
9284.604268643, 9787.525726881, 9033.316521658, 9469.444921654, 8726.555858861, 8905.098118171,
8437.173060618, 8935.043125766, 9172.334141304, 8425.564554151, 8725.52174345, 8457.038873118,
8672.538442695, 11321.501722433, 8233.9854963, 9465.167737138, 8841.407418574, 8476.337700492,
9162.816556878, 9946.670992861, 9281.411111594, 8349.695093959, 8671.731748701, 8611.662470696,
9074.152941559, 8296.712958538, 9208.592940681, 8041.287735389, 9707.228511359 9284.604268643,
9787.525726881,
9033.316521658,
9469.444921654,
8726.555858861,
8437.173060618,
8935.043125766,
9172.334141304,
8425.564554151,
8725.52174345,
8672.538442695,
11321.501722433,
8233.9854963,
9465.167737138,
8841.407418574,
9162.816556878,
9946.670992861,
9281.411111594,
8349.695093959,
8671.731748701,
9074.152941559,
8296.712958538,
9208.592940681,
8041.287735389,
9707.228511359,
] ]
scenario3_profits = [ scenario3_profits = [
9268.899026395, 9430.210649623, 8919.631184021, 8944.062758608, 9095.775246456, 9268.899026395,
9886.312580673, 10387.393647889, 9637.953603446, 10098.221535588, 9238.913391815, 9430.210649623,
8980.977156104, 9414.393022001, 9722.356857713, 8917.726783311, 9178.474614142, 8919.631184021,
9167.292074295, 12011.512542804, 8693.048121325, 10039.47692674, 9375.735645214, 8944.062758608,
9730.36014247, 10531.376234688, 9750.101324443, 8881.341453044, 9186.289888284, 9095.775246456,
9675.754350507, 8773.329747496, 9722.523537949, 8489.277170975, 10325.425491734 9886.312580673,
10387.393647889,
9637.953603446,
10098.221535588,
9238.913391815,
8980.977156104,
9414.393022001,
9722.356857713,
8917.726783311,
9178.474614142,
9167.292074295,
12011.512542804,
8693.048121325,
10039.47692674,
9375.735645214,
9730.36014247,
10531.376234688,
9750.101324443,
8881.341453044,
9186.289888284,
9675.754350507,
8773.329747496,
9722.523537949,
8489.277170975,
10325.425491734,
] ]
# Create figure with single plot # Create figure with single plot
@ -42,17 +130,23 @@ num_points = len(scenario1_profits)
y_values = [i / num_points for i in range(1, num_points + 1)] y_values = [i / num_points for i in range(1, num_points + 1)]
# Plot all three scenarios on the same graph # Plot all three scenarios on the same graph
plt.plot(scenario1_sorted, y_values, 'b-o', label='Scenariusz 1', linewidth=2, markersize=4) plt.plot(
plt.plot(scenario2_sorted, y_values, 'g-s', label='Scenariusz 2', linewidth=2, markersize=4) scenario1_sorted, y_values, "b-o", label="Scenariusz 1", linewidth=2, markersize=4
plt.plot(scenario3_sorted, y_values, 'r-^', label='Scenariusz 3', linewidth=2, markersize=4) )
plt.plot(
scenario2_sorted, y_values, "g-s", label="Scenariusz 2", linewidth=2, markersize=4
)
plt.plot(
scenario3_sorted, y_values, "r-^", label="Scenariusz 3", linewidth=2, markersize=4
)
# Add labels and title # Add labels and title
plt.xlabel('Przeciętny Zysk', fontsize=16) plt.xlabel("Przeciętny Zysk", fontsize=LABEL_FONT_SIZE)
plt.ylabel('Odwrotna dystrybuanta', fontsize=16) plt.ylabel("Odwrotna dystrybuanta", fontsize=LABEL_FONT_SIZE)
plt.xticks(fontsize=16) plt.xticks(fontsize=LABEL_FONT_SIZE)
plt.yticks(fontsize=16) plt.yticks(fontsize=LABEL_FONT_SIZE)
plt.ylim(0, 1) # Set y-axis limits to cut off values outside 0-1 range plt.ylim(0, 1) # Set y-axis limits to cut off values outside 0-1 range
plt.grid(True, alpha=0.3) plt.grid(visible=True, alpha=0.3)
plt.legend(fontsize=16) plt.legend(fontsize=LABEL_FONT_SIZE)
# Format the plot # Format the plot
plt.tight_layout() plt.tight_layout()
@ -61,7 +155,13 @@ plt.tight_layout()
plt.show() plt.show()
# Print statistics for each scenario # Print statistics for each scenario
print("Scenario Statistics:") logger.info("Scenario Statistics:")
print(f"Scenario 1 - Mean: {np.mean(scenario1_profits):.2f}, Std: {np.std(scenario1_profits):.2f}") for number, profits in enumerate(
print(f"Scenario 2 - Mean: {np.mean(scenario2_profits):.2f}, Std: {np.std(scenario2_profits):.2f}") (scenario1_profits, scenario2_profits, scenario3_profits), start=1
print(f"Scenario 3 - Mean: {np.mean(scenario3_profits):.2f}, Std: {np.std(scenario3_profits):.2f}") ):
logger.info(
"Scenario %d - Mean: %.2f, Std: %.2f",
number,
np.mean(profits),
np.std(profits),
)

153
Programming/WDWR/projekt/graph3.py Normal file → Executable file
View File

@ -1,23 +1,120 @@
#!/usr/bin/env python3
"""Plot the inverse distribution of RISK for the three WDWR scenarios.
The companion of graph2.py: same three scenarios, same empirical cumulative
probability on the y axis, but the Gini risk measure on the x axis instead of
the profit.
"""
import logging
import matplotlib.pyplot as plt import matplotlib.pyplot as plt
import numpy as np import numpy as np
logger = logging.getLogger(__name__)
logging.basicConfig(format="%(message)s", level=logging.INFO)
# Sized for a report figure rather than for the screen.
LABEL_FONT_SIZE = 16
# Data for the three scenarios - Risk values # Data for the three scenarios - Risk values
scenario1_risks = [ scenario1_risks = [
184.458, 195.09, 207.678, 278.26, 220.823, 181.042, 195.281, 199.255, 182.194, 181.295, 184.458,
285.773, 185.306, 272.997, 288.405, 223.407, 213.341, 208.693, 354.01, 217.093, 301.892, 195.09,
200.499, 637.424, 408.232, 308.314, 181.042, 470.693, 212.039, 184.694, 345.455, 205.291 207.678,
278.26,
220.823,
181.042,
195.281,
199.255,
182.194,
181.295,
285.773,
185.306,
272.997,
288.405,
223.407,
213.341,
208.693,
354.01,
217.093,
301.892,
200.499,
637.424,
408.232,
308.314,
181.042,
470.693,
212.039,
184.694,
345.455,
205.291,
] ]
scenario2_risks = [ scenario2_risks = [
403.784, 414.043, 408.225, 664.379, 413.447, 448.303, 513.763, 439.966, 403.784, 408.366, 403.784,
773.926, 459.531, 537.056, 795.835, 542.569, 512.817, 471.741, 879.425, 441.46, 533.521, 414.043,
659.742, 990.73, 842.758, 776.278, 404.787, 989.35, 481.833, 409.978, 795.275, 421.261 408.225,
664.379,
413.447,
448.303,
513.763,
439.966,
403.784,
408.366,
773.926,
459.531,
537.056,
795.835,
542.569,
512.817,
471.741,
879.425,
441.46,
533.521,
659.742,
990.73,
842.758,
776.278,
404.787,
989.35,
481.833,
409.978,
795.275,
421.261,
] ]
scenario3_risks = [ scenario3_risks = [
559.839, 576.633, 596.264, 942.092, 560.045, 679.026, 683.47, 627.607, 559.839, 562.478, 559.839,
1245.41, 638.37, 675.366, 1132.8, 784.609, 686.634, 680.842, 1115.92, 572.793, 682.114, 576.633,
914.647, 1211.67, 1051.95, 1104.87, 583.36, 1360.71, 743.425, 577.734, 1191.09, 581.791 596.264,
942.092,
560.045,
679.026,
683.47,
627.607,
559.839,
562.478,
1245.41,
638.37,
675.366,
1132.8,
784.609,
686.634,
680.842,
1115.92,
572.793,
682.114,
914.647,
1211.67,
1051.95,
1104.87,
583.36,
1360.71,
743.425,
577.734,
1191.09,
581.791,
] ]
# Create figure with single plot # Create figure with single plot
@ -33,18 +130,24 @@ num_points = len(scenario1_risks)
y_values = [i / num_points for i in range(1, num_points + 1)] y_values = [i / num_points for i in range(1, num_points + 1)]
# Plot all three scenarios on the same graph # Plot all three scenarios on the same graph
plt.plot(scenario1_sorted, y_values, 'b-o', label='Scenariusz 1', linewidth=2, markersize=4) plt.plot(
plt.plot(scenario2_sorted, y_values, 'g-s', label='Scenariusz 2', linewidth=2, markersize=4) scenario1_sorted, y_values, "b-o", label="Scenariusz 1", linewidth=2, markersize=4
plt.plot(scenario3_sorted, y_values, 'r-^', label='Scenariusz 3', linewidth=2, markersize=4) )
plt.plot(
scenario2_sorted, y_values, "g-s", label="Scenariusz 2", linewidth=2, markersize=4
)
plt.plot(
scenario3_sorted, y_values, "r-^", label="Scenariusz 3", linewidth=2, markersize=4
)
# Add labels and title # Add labels and title
plt.xlabel('Ryzyko', fontsize=16) plt.xlabel("Ryzyko", fontsize=LABEL_FONT_SIZE)
plt.ylabel('Odwrotna dystrybuanta', fontsize=16) plt.ylabel("Odwrotna dystrybuanta", fontsize=LABEL_FONT_SIZE)
plt.xticks(fontsize=16) plt.xticks(fontsize=LABEL_FONT_SIZE)
plt.yticks(fontsize=16) plt.yticks(fontsize=LABEL_FONT_SIZE)
plt.ylim(0, 1) # Set y-axis limits to cut off values outside 0-1 range plt.ylim(0, 1) # Set y-axis limits to cut off values outside 0-1 range
plt.grid(True, alpha=0.3) plt.grid(visible=True, alpha=0.3)
plt.legend(fontsize=16) plt.legend(fontsize=LABEL_FONT_SIZE)
# Format the plot # Format the plot
plt.tight_layout() plt.tight_layout()
@ -53,10 +156,10 @@ plt.tight_layout()
plt.show() plt.show()
# Print statistics for each scenario # Print statistics for each scenario
print("Scenario Statistics:") logger.info("Scenario Statistics:")
print(f"Scenario 1 - Mean: {np.mean(scenario1_risks):.2f}, " for number, risks in enumerate(
f"Std: {np.std(scenario1_risks):.2f}") (scenario1_risks, scenario2_risks, scenario3_risks), start=1
print(f"Scenario 2 - Mean: {np.mean(scenario2_risks):.2f}, " ):
f"Std: {np.std(scenario2_risks):.2f}") logger.info(
print(f"Scenario 3 - Mean: {np.mean(scenario3_risks):.2f}, " "Scenario %d - Mean: %.2f, Std: %.2f", number, np.mean(risks), np.std(risks)
f"Std: {np.std(scenario3_risks):.2f}") )

949
Programming/twm_4/TWM_KerasIntro.ipynb
View File

File diff suppressed because it is too large Load Diff

302
Programming/twm_4/TWM_KerasIntro.py
View File

@ -4,75 +4,88 @@
# In[ ]: # In[ ]:
import sys import sys
import tensorflow as tf import tensorflow as tf
# Check if GPU is available # Check if GPU is available
print(tf.config.list_physical_devices('GPU')) print(tf.config.list_physical_devices("GPU"))
if tf.config.list_physical_devices('GPU'): if tf.config.list_physical_devices("GPU"):
print("GPU is available") print("GPU is available")
else: else:
print("GPU is not available") print("GPU is not available")
sys.exit() sys.exit()
from tensorflow import keras
from keras.datasets import cifar10
from keras.utils import to_categorical
import numpy as np
import itertools import itertools
import matplotlib.pyplot as plt
from keras.models import Sequential
from sklearn.metrics import confusion_matrix
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from keras.layers import BatchNormalization, Conv2D, MaxPooling2D, ZeroPadding2D, GlobalAveragePooling2D, Flatten, Dense, Dropout, Activation
from tensorflow.keras.optimizers import Adam
def plot_confusion_matrix(cm, classes, import matplotlib.pyplot as plt
normalize=False, import numpy as np
title='Confusion matrix', from keras.datasets import cifar10
cmap=plt.cm.Blues): from keras.layers import (
""" Activation,
This function prints and plots the confusion matrix. BatchNormalization,
Conv2D,
Dense,
Dropout,
Flatten,
MaxPooling2D,
)
from keras.models import Sequential
from keras.utils import to_categorical
from sklearn.metrics import confusion_matrix
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.preprocessing.image import ImageDataGenerator
def plot_confusion_matrix(
cm, classes, normalize=False, title="Confusion matrix", cmap=plt.cm.Blues
):
"""This function prints and plots the confusion matrix.
Normalization can be applied by setting `normalize=True`. Normalization can be applied by setting `normalize=True`.
""" """
if normalize: if normalize:
cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis] cm = cm.astype("float") / cm.sum(axis=1)[:, np.newaxis]
print("Normalized confusion matrix") print("Normalized confusion matrix")
else: else:
print('Confusion matrix, without normalization') print("Confusion matrix, without normalization")
print(cm) print(cm)
plt.imshow(cm, interpolation="nearest", cmap=cmap)
plt.imshow(cm, interpolation='nearest', cmap=cmap)
plt.title(title) plt.title(title)
plt.colorbar() plt.colorbar()
tick_marks = np.arange(len(classes)) tick_marks = np.arange(len(classes))
plt.xticks(tick_marks, classes, rotation=45) plt.xticks(tick_marks, classes, rotation=45)
plt.yticks(tick_marks, classes) plt.yticks(tick_marks, classes)
fmt = '.2f' if normalize else 'd' fmt = ".2f" if normalize else "d"
thresh = cm.max() / 2. thresh = cm.max() / 2.0
for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])): for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
plt.text(j, i, format(cm[i, j], fmt), plt.text(
horizontalalignment="center", j,
color="white" if cm[i, j] > thresh else "black") i,
format(cm[i, j], fmt),
horizontalalignment="center",
color="white" if cm[i, j] > thresh else "black",
)
plt.ylabel('True label') plt.ylabel("True label")
plt.xlabel('Predicted label') plt.xlabel("Predicted label")
plt.tight_layout() plt.tight_layout()
(X_train, y_train), (X_test, y_test) = cifar10.load_data() (X_train, y_train), (X_test, y_test) = cifar10.load_data()
X_train = X_train.astype('float32') # change integers to 32-bit floating point numbers X_train = X_train.astype("float32") # change integers to 32-bit floating point numbers
X_test = X_test.astype('float32') X_test = X_test.astype("float32")
X_train /= 255 # normalize each value for each pixel for the entire vector for each input X_train /= (
255 # normalize each value for each pixel for the entire vector for each input
)
X_test /= 255 X_test /= 255
y_train = y_train.reshape((1,-1))[0] y_train = y_train.reshape((1, -1))[0]
y_test = y_test.reshape((1,-1))[0] y_test = y_test.reshape((1, -1))[0]
print("Training matrix shape", X_train.shape, y_train.shape) print("Training matrix shape", X_train.shape, y_train.shape)
print("Testing matrix shape", X_test.shape, y_test.shape) print("Testing matrix shape", X_test.shape, y_test.shape)
@ -84,7 +97,18 @@ nb_classes = 10
Y_train = to_categorical(y_train, nb_classes) Y_train = to_categorical(y_train, nb_classes)
Y_test = to_categorical(y_test, nb_classes) Y_test = to_categorical(y_test, nb_classes)
cifar_names = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'] cifar_names = [
"airplane",
"automobile",
"bird",
"cat",
"deer",
"dog",
"frog",
"horse",
"ship",
"truck",
]
# ## Podgląd zbioru treningowego # ## Podgląd zbioru treningowego
@ -92,16 +116,23 @@ cifar_names = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', '
# In[ ]: # In[ ]:
for i in range(0, 10): for i in range(10):
img_batch = X_train[y_train == i][0:10] img_batch = X_train[y_train == i][0:10]
img_batch = np.reshape(img_batch, (img_batch.shape[0]*img_batch.shape[1], img_batch.shape[2], img_batch.shape[3])) img_batch = np.reshape(
if i > 0: img_batch,
img = np.concatenate([img, img_batch], axis = 1) (
else: img_batch.shape[0] * img_batch.shape[1],
img = img_batch img_batch.shape[2],
plt.figure(figsize=(10,20)) img_batch.shape[3],
plt.axis('off') ),
plt.imshow(img, cmap='gray') )
if i > 0:
img = np.concatenate([img, img_batch], axis=1)
else:
img = img_batch
plt.figure(figsize=(10, 20))
plt.axis("off")
plt.imshow(img, cmap="gray")
# ## Przygotowanie modelu # ## Przygotowanie modelu
@ -110,98 +141,103 @@ plt.imshow(img, cmap='gray')
def generate_model(): def generate_model():
model = Sequential() # Linear stacking of layers model = Sequential() # Linear stacking of layers
# Convolution Layer 1 # Convolution Layer 1
model.add(Conv2D(16, (3, 3), input_shape=(32,32,3))) model.add(Conv2D(16, (3, 3), input_shape=(32, 32, 3)))
model.add(Activation('relu') ) model.add(Activation("relu"))
# ... # ...
model.add(Flatten()) # Flatten final output matrix into a vector model.add(Flatten()) # Flatten final output matrix into a vector
# ... # ...
# Fully Connected Layer # Fully Connected Layer
model.add(Dense(10)) # final 10 FC nodes model.add(Dense(10)) # final 10 FC nodes
model.add(Activation('softmax')) # softmax activation model.add(Activation("softmax")) # softmax activation
model.summary() model.summary()
adam = tf.optimizers.Adam(learning_rate=0.001) adam = tf.optimizers.Adam(learning_rate=0.001)
model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy']) model.compile(loss="categorical_crossentropy", optimizer=adam, metrics=["accuracy"])
return model
return model
def generate_model_default(): def generate_model_default():
model = Sequential() # Linear stacking of layers model = Sequential() # Linear stacking of layers
# Convolution Layer 1 # Convolution Layer 1
model.add(Conv2D(16, (3, 3), input_shape=(32,32,3))) model.add(Conv2D(16, (3, 3), input_shape=(32, 32, 3)))
model.add(Activation('relu') ) model.add(Activation("relu"))
# ... # ...
model.add(Flatten()) # Flatten final output matrix into a vector model.add(Flatten()) # Flatten final output matrix into a vector
# ... # ...
# Fully Connected Layer # Fully Connected Layer
model.add(Dense(10)) # final 10 FC nodes model.add(Dense(10)) # final 10 FC nodes
model.add(Activation('softmax')) # softmax activation model.add(Activation("softmax")) # softmax activation
model.summary() model.summary()
adam = tf.optimizers.Adam(learning_rate=0.001) adam = tf.optimizers.Adam(learning_rate=0.001)
model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy']) model.compile(loss="categorical_crossentropy", optimizer=adam, metrics=["accuracy"])
return model
return model
def generate_model_gemini(): def generate_model_gemini():
model = Sequential() model = Sequential()
# Convolutional Layers with Max Pooling # Convolutional Layers with Max Pooling
model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3))) model.add(Conv2D(32, (3, 3), activation="relu", input_shape=(32, 32, 3)))
model.add(MaxPooling2D(pool_size=(2, 2))) model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(64, (3, 3), activation='relu')) model.add(Conv2D(64, (3, 3), activation="relu"))
model.add(MaxPooling2D(pool_size=(2, 2))) model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25)) # Regularization model.add(Dropout(0.25)) # Regularization
# Flatten and Fully Connected Layers # Flatten and Fully Connected Layers
model.add(Flatten()) model.add(Flatten())
model.add(Dense(128, activation='relu')) model.add(Dense(128, activation="relu"))
model.add(Dropout(0.5)) # Regularization model.add(Dropout(0.5)) # Regularization
model.add(Dense(10, activation='softmax')) model.add(Dense(10, activation="softmax"))
# Model Compilation # Model Compilation
model.compile(loss='categorical_crossentropy', model.compile(
optimizer='adam', # Consider trying other optimizers loss="categorical_crossentropy",
metrics=['accuracy']) optimizer="adam", # Consider trying other optimizers
metrics=["accuracy"],
)
return model return model
def generate_model_chat(): def generate_model_chat():
model = Sequential() # Linear stacking of layers model = Sequential() # Linear stacking of layers
# Convolution Layer 1 # Convolution Layer 1
model.add(Conv2D(32, (3, 3), padding='same', input_shape=(32, 32, 3))) model.add(Conv2D(32, (3, 3), padding="same", input_shape=(32, 32, 3)))
model.add(Activation('relu')) model.add(Activation("relu"))
model.add(BatchNormalization()) model.add(BatchNormalization())
# Convolution Layer 2 # Convolution Layer 2
model.add(Conv2D(32, (3, 3))) model.add(Conv2D(32, (3, 3)))
model.add(Activation('relu')) model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2))) model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.3)) model.add(Dropout(0.3))
# Convolution Layer 3 # Convolution Layer 3
model.add(Conv2D(64, (3, 3), padding='same')) model.add(Conv2D(64, (3, 3), padding="same"))
model.add(Activation('relu')) model.add(Activation("relu"))
model.add(BatchNormalization()) model.add(BatchNormalization())
# Convolution Layer 4 # Convolution Layer 4
model.add(Conv2D(64, (3, 3))) model.add(Conv2D(64, (3, 3)))
model.add(Activation('relu')) model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2))) model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.3)) model.add(Dropout(0.3))
@ -209,20 +245,20 @@ def generate_model_chat():
model.add(Flatten()) model.add(Flatten())
# Fully Connected Layer # Fully Connected Layer
model.add(Dense(512)) # Large fully connected layer model.add(Dense(512)) # Large fully connected layer
model.add(Activation('relu')) model.add(Activation("relu"))
model.add(BatchNormalization()) model.add(BatchNormalization())
model.add(Dropout(0.6)) model.add(Dropout(0.6))
# Output Layer # Output Layer
model.add(Dense(10)) # final 10 FC nodes model.add(Dense(10)) # final 10 FC nodes
model.add(Activation('softmax')) # softmax activation model.add(Activation("softmax")) # softmax activation
model.summary() model.summary()
# Compile the model # Compile the model
adam = Adam(learning_rate=0.001) adam = Adam(learning_rate=0.001)
model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy']) model.compile(loss="categorical_crossentropy", optimizer=adam, metrics=["accuracy"])
return model return model
@ -242,11 +278,17 @@ models = [model_chat]
# In[ ]: # In[ ]:
gen = ImageDataGenerator(rotation_range=8, width_shift_range=0.08, shear_range=0.3, gen = ImageDataGenerator(
height_shift_range=0.08, zoom_range=0.08, validation_split=0.2) rotation_range=8,
width_shift_range=0.08,
shear_range=0.3,
height_shift_range=0.08,
zoom_range=0.08,
validation_split=0.2,
)
train_generator = gen.flow(X_train, Y_train, batch_size=128, subset='training') train_generator = gen.flow(X_train, Y_train, batch_size=128, subset="training")
valid_generator = gen.flow(X_train, Y_train, batch_size=128, subset='validation') valid_generator = gen.flow(X_train, Y_train, batch_size=128, subset="validation")
# In[ ]: # In[ ]:
@ -254,7 +296,14 @@ valid_generator = gen.flow(X_train, Y_train, batch_size=128, subset='validation'
# Max 20 epoch # Max 20 epoch
for model in models: for model in models:
model.fit(train_generator, steps_per_epoch=40000//128, epochs=20, verbose=1, validation_data=valid_generator, validation_steps = 10000 // 128) model.fit(
train_generator,
steps_per_epoch=40000 // 128,
epochs=20,
verbose=1,
validation_data=valid_generator,
validation_steps=10000 // 128,
)
# ## Test # ## Test
@ -262,50 +311,54 @@ for model in models:
# In[ ]: # In[ ]:
for model in models: for model in models:
score = model.evaluate(X_test, Y_test) score = model.evaluate(X_test, Y_test)
print('Test score:', score[0]) print("Test score:", score[0])
print('Test accuracy:', score[1]) print("Test accuracy:", score[1])
# The predict_classes function outputs the highest probability class # The predict_classes function outputs the highest probability class
# according to the trained classifier for each input example. # according to the trained classifier for each input example.
predicted = model.predict(X_test) predicted = model.predict(X_test)
predicted_classes = np.argmax(predicted, axis=1) predicted_classes = np.argmax(predicted, axis=1)
# Check which items we got right / wrong # Check which items we got right / wrong
correct_indices = np.nonzero(predicted_classes == y_test)[0] correct_indices = np.nonzero(predicted_classes == y_test)[0]
incorrect_indices = np.nonzero(predicted_classes != y_test)[0] incorrect_indices = np.nonzero(predicted_classes != y_test)[0]
cnf_matrix = confusion_matrix(y_test, predicted_classes)
cnf_matrix = confusion_matrix(y_test, predicted_classes) class_names = [str(i) for i in range(10)]
class_names = [str(i) for i in range(10)] # Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(
cnf_matrix, classes=class_names, title="Confusion matrix, without normalization"
)
# Plot non-normalized confusion matrix plt.show()
plt.figure()
plot_confusion_matrix(cnf_matrix, classes=class_names,
title='Confusion matrix, without normalization')
plt.show()
# In[ ]: # In[ ]:
def show_samples_rgb(indices, preds, images, labels, count=3, names = []): def show_samples_rgb(indices, preds, images, labels, count=3, names=[]):
plt.figure() plt.figure()
for i, sample in enumerate(indices[:count**2]): for i, sample in enumerate(indices[: count**2]):
pred_id = int(np.argmax(preds[sample])) pred_id = int(np.argmax(preds[sample]))
real_id = int(labels[sample]) real_id = int(labels[sample])
pred_score = preds[sample][pred_id] pred_score = preds[sample][pred_id]
real_score = preds[sample][real_id] real_score = preds[sample][real_id]
plt.subplot(count,count,i+1) plt.subplot(count, count, i + 1)
plt.imshow(images[sample], interpolation='none') plt.imshow(images[sample], interpolation="none")
plt.axis('off') plt.axis("off")
if len(names) > 0: if len(names) > 0:
plt.title("P: {} ({:.2f})\nE: {} ({:.2f})".format(names[pred_id], pred_score, names[real_id], real_score)) plt.title(
f"P: {names[pred_id]} ({pred_score:.2f})\nE: {names[real_id]} ({real_score:.2f})"
)
else: else:
plt.title("P: {} ({:.2f})\nE: {} ({:.2f})".format(pred_id, pred_score, real_id, real_score)) plt.title(
f"P: {pred_id} ({pred_score:.2f})\nE: {real_id} ({real_score:.2f})"
)
plt.tight_layout() plt.tight_layout()
@ -324,4 +377,3 @@ show_samples_rgb(correct_indices, predicted, X_test, y_test, 5, cifar_names)
show_samples_rgb(incorrect_indices, predicted, X_test, y_test, 5, cifar_names) show_samples_rgb(incorrect_indices, predicted, X_test, y_test, 5, cifar_names)

16
Programming/twm_4/twm.py
View File

@ -1,19 +1,3 @@
import tensorflow as tf import tensorflow as tf
import numpy as np # advanced math library
import matplotlib.pyplot as plt # MATLAB like plotting routines
import random # for generating random numbers
from keras.datasets import mnist # MNIST dataset is included in Keras%
from keras.models import Sequential # Model type to be used
from keras.layers import Dense, Dropout, Activation # Types of layers to be used in our model
from keras.utils import to_categorical # NumPy related tools
from keras import optimizers
from sklearn.metrics import confusion_matrix
import itertools
print(tf.__version__) print(tf.__version__)

2
gates/runnability-local.tsv
View File

@ -37,7 +37,7 @@ Programming/PBAD blocked jupyter is on PATH but broken (bad interpreter)
Programming/PORR runnable python3 /home/kuhy/WUT_Computer_Science/Programming/PORR/code/main.py Programming/PORR runnable python3 /home/kuhy/WUT_Computer_Science/Programming/PORR/code/main.py
Programming/PSD runnable starts the Kafka/Flink stack (installs packages with sudo) Programming/PSD runnable starts the Kafka/Flink stack (installs packages with sudo)
Programming/psd_project runnable mvn package Programming/psd_project runnable mvn package
Programming/TRAK runnable python3 the entry script Programming/TRAK runnable python3 /home/kuhy/WUT_Computer_Science/Programming/TRAK/main.py
Programming/twm_4 blocked jupyter is on PATH but broken (bad interpreter) Programming/twm_4 blocked jupyter is on PATH but broken (bad interpreter)
Programming/USD no-code README describes a MetaDrive RL assignment; no implementation was submitted Programming/USD no-code README describes a MetaDrive RL assignment; no implementation was submitted
Programming/WDWR runnable python3 the entry script Programming/WDWR runnable python3 the entry script

1 NotProgramming/DPZ runnable builds the report with latexmk
37 Programming/PORR runnable python3 /home/kuhy/WUT_Computer_Science/Programming/PORR/code/main.py
38 Programming/PSD runnable starts the Kafka/Flink stack (installs packages with sudo)
39 Programming/psd_project runnable mvn package
40 Programming/TRAK runnable python3 the entry script python3 /home/kuhy/WUT_Computer_Science/Programming/TRAK/main.py
41 Programming/twm_4 blocked jupyter is on PATH but broken (bad interpreter)
42 Programming/USD no-code README describes a MetaDrive RL assignment; no implementation was submitted
43 Programming/WDWR runnable python3 the entry script

19
gates/vendored.txt
View File

@ -62,3 +62,22 @@ NotProgramming/EMISY/schematic
# ControlPanel.java (1852 lines) and Kernel.java (511) are not split. # ControlPanel.java (1852 lines) and Kernel.java (511) are not split.
Programming/EOPSY/lab3/task3/work Programming/EOPSY/lab3/task3/work
Programming/EOPSY/lab4/task4/work Programming/EOPSY/lab4/task4/work
# --- Third-party ray tracer ------------------------------------------------
# sightpy, the rendering engine from rafael-fuente/Python-Raytracer (BSD
# 3-Clause, (c) 2021 Rafael de la Fuente). TRAK's assignment was photon
# mapping ON TOP of it; the engine itself was not written here.
#
# Verified 2026-08-28 against a fresh clone of the upstream repository, after
# normalising line endings (the copy here is LF, upstream ships CRLF):
# 30 of 32 .py files BYTE-IDENTICAL
# sightpy/ray.py +12 lines: RAYS / PRIMARY_RAYS counters and get_rays()
# sightpy/scene.py +13 lines: prints those counters in the render summary
# Both patches are kuhy's instrumentation for the report's ray statistics and
# are the only authored lines inside this tree.
#
# NOT covered by this entry, and still fully gated: TRAK's own main.py,
# photon_mapping.py, rendering.py, lab6.py, utils.py, code/ and scenes/.
# scenes/ is adapted from the upstream examples rather than copied -- the
# closest example differs by more lines than the scene file has.
Programming/TRAK/sightpy

93
pyproject.toml Normal file
View File

@ -0,0 +1,93 @@
# Python tooling for the whole archive. There is no package here to build --
# this file exists so ruff, mypy and coverage read one configuration instead of
# 25 course directories each guessing.
#
# The `exclude` and `omit` lists below are MIRRORS of gates/vendored.txt.
# Edit that manifest and re-run `./run.sh sync-vendored`; never edit a mirror
# on its own, or the gates start disagreeing about what is kuhy's code.
[tool.ruff]
target-version = "py310"
line-length = 88
include = ["*.py", "**/*.py", "*.ipynb", "**/*.ipynb"]
exclude = [
# --- machine-generated, never authored ---
".git",
".venv",
"__pycache__",
"build",
"dist",
"node_modules",
# --- mirrors of gates/vendored.txt ---
"Programming/EGUI/lab2/lab2/wwwroot/lib",
"Programming/E-PSYOPS/theProject/Assets/TextMesh Pro",
"Programming/E-PSYOPS/theProject/Assets/TutorialInfo",
"NotProgramming/EMISY/componentSchematic",
"NotProgramming/EMISY/schematic",
"Programming/EOPSY/lab3/task3/work",
"Programming/EOPSY/lab4/task4/work",
"Programming/TRAK/sightpy",
]
[tool.ruff.lint]
select = ["ALL"]
# The only rules turned off, and why. There are no per-file-ignores: the
# coursework is held to the same bar as the tooling, because git history is
# what preserves the version that was handed in.
ignore = [
# Pairwise-contradictory docstring rules -- ruff ships both halves of each
# pair in ALL and you must pick one. These match the rest of the fleet.
"D203", # ...vs D211, we take D211 (no blank line before class docstring)
"D213", # ...vs D212, we take D212 (summary on the first line)
# Disabled on the formatter's own advice: the formatter already decides
# both of these, and leaving them on makes the two tools fight.
# https://docs.astral.sh/ruff/formatter/#conflicting-lint-rules
"COM812", # missing trailing comma
"ISC001", # single-line implicit string concatenation
# A known false-positive generator: it cannot see that an argument list was
# validated upstream. https://github.com/astral-sh/ruff/issues/4045
"S603", # subprocess call without shell=True
# Ruff began enforcing this outside preview in 0.16; no repo in this fleet
# carries copyright headers, and stamping one onto 137 university
# assignments would be asserting a licence that was never claimed.
"CPY001", # missing copyright notice
]
fixable = ["ALL"]
unfixable = []
[tool.mypy]
strict = true
exclude = [
"^\\.venv/",
"^build/",
"^Programming/EGUI/lab2/lab2/wwwroot/lib/",
"^Programming/E-PSYOPS/theProject/Assets/TextMesh Pro/",
"^Programming/E-PSYOPS/theProject/Assets/TutorialInfo/",
"^NotProgramming/EMISY/componentSchematic/",
"^NotProgramming/EMISY/schematic/",
"^Programming/EOPSY/lab3/task3/work/",
"^Programming/EOPSY/lab4/task4/work/",
"^Programming/TRAK/sightpy/",
]
[tool.coverage.run]
branch = true
omit = [
"*/.venv/*",
"*/build/*",
"Programming/EGUI/lab2/lab2/wwwroot/lib/*",
"Programming/E-PSYOPS/theProject/Assets/TextMesh Pro/*",
"Programming/E-PSYOPS/theProject/Assets/TutorialInfo/*",
"NotProgramming/EMISY/componentSchematic/*",
"NotProgramming/EMISY/schematic/*",
"Programming/EOPSY/lab3/task3/work/*",
"Programming/EOPSY/lab4/task4/work/*",
"Programming/TRAK/sightpy/*",
]
[tool.coverage.report]
show_missing = true
skip_covered = false

7
run.sh
View File

@ -153,6 +153,13 @@ cmd_run() {
local runner="${SCRIPT_DIR}/${target}/run.sh" local runner="${SCRIPT_DIR}/${target}/run.sh"
[[ -x "$runner" ]] || { err "no executable run.sh in $target"; return 1; } [[ -x "$runner" ]] || { err "no executable run.sh in $target"; return 1; }
log "running $target (capped at ${RUN_MEMORY_MAX} RAM, ${RUN_CPU_QUOTA} CPU)" log "running $target (capped at ${RUN_MEMORY_MAX} RAM, ${RUN_CPU_QUOTA} CPU)"
# Agg, so plt.show() draws nothing and returns. TRAK's renderer finished
# its ray trace in a couple of minutes and then sat there for 45 more:
# plt.show() had opened an image viewer and was waiting for a human to
# close the window. A run that needs a person at a keyboard is not a run.
# export, not an assignment prefix: _run_capped is a shell function, and a
# prefixed assignment on one is not put into the child's environment.
export MPLBACKEND="${MPLBACKEND:-Agg}"
_run_capped "$runner" _run_capped "$runner"
} }

261
scripts/course_lib.sh
View File

@ -11,7 +11,8 @@
# (no args) actually run it # (no args) actually run it
# #
# A stub calls: # A stub calls:
# course_main --kind <kind> [--entry <path>] [--note <text>] -- "$@" # course_main --kind <kind> [--entry <path>] [--entry-args <str>]
# [--note <text>] [--test-dir <path>] [--no-bibtex] -- "$@"
# Values are passed as arguments rather than set as globals so that a stub has # Values are passed as arguments rather than set as globals so that a stub has
# no assignments a linter has to be told to ignore. # no assignments a linter has to be told to ignore.
@ -80,252 +81,18 @@ _first_file() {
find "$COURSE_ROOT" -name "$1" -not -path '*/node_modules/*' -print -quit 2>/dev/null find "$COURSE_ROOT" -name "$1" -not -path '*/node_modules/*' -print -quit 2>/dev/null
} }
# --- probes ------------------------------------------------------------------
_probe() { # The probe and the runner are one case statement each and together are most of
case "$COURSE_KIND" in # this file's length, so they live beside it rather than in it. SCRIPTDIR is
none) _no_code "${COURSE_NOTE:-no executable code in this directory}" ;; # resolved from BASH_SOURCE[0] -- BASH_SOURCE[1] is the *caller*, which is a
tex) # per-course run.sh two directories away.
_probe_any "a LaTeX toolchain" latexmk pdflatex xelatex _COURSE_LIB_DIR="$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd)"
_runnable "builds ${COURSE_ENTRY:-the report} with latexmk" # shellcheck source-path=SCRIPTDIR
;; # shellcheck source=course_probe.sh
python) . "${_COURSE_LIB_DIR}/course_probe.sh"
_have python3 || _blocked "needs python3" # shellcheck source-path=SCRIPTDIR
[[ -n "$(_first_file '*.py')" ]] || _no_code "no .py files found" # shellcheck source=course_run.sh
_runnable "python3 ${COURSE_ENTRY:-the entry script}" . "${_COURSE_LIB_DIR}/course_run.sh"
;;
node)
_have node || _blocked "needs node"
_have npm || _blocked "needs npm"
_runnable "npm install && npm start"
;;
maven)
_have mvn || _blocked "needs maven"
_have java || _blocked "needs a JDK"
_runnable "mvn package"
;;
make)
_have make || _blocked "needs make"
_probe_any "a C compiler" gcc cc clang
_runnable "make"
;;
dotnet)
_have dotnet || _blocked "needs the .NET SDK"
_runnable "dotnet build"
;;
unity)
_have unity-editor || _have unityhub \
|| _blocked "needs the Unity editor (GUI, not installable on a CI runner)"
_runnable "opens in the Unity editor"
;;
matlab)
_probe_any "MATLAB or Octave" matlab octave
_runnable "runs the .m scripts"
;;
cxx)
_probe_any "a C++ compiler" g++ clang++
[[ -n "$(_first_file '*.cpp')" ]] || _no_code "no .cpp files found"
_runnable "compiles each standalone .cpp"
;;
notebook)
_have jupyter || _blocked "needs jupyter"
# Being on PATH is not the same as working: this machine has a
# jupyter shim whose pipx venv interpreter is gone, so it fails
# with "bad interpreter". Probe by actually invoking it.
jupyter --version > /dev/null 2>&1 || _blocked "jupyter is on PATH but broken (bad interpreter)"
_runnable "jupyter nbconvert --execute"
;;
compose)
_have docker || _blocked "needs docker"
docker info > /dev/null 2>&1 || _blocked "docker is installed but the daemon is not running"
_runnable "docker compose up"
;;
*) _blocked "unknown COURSE_KIND '${COURSE_KIND:-unset}'" ;;
esac
}
# Use the course venv when one exists, otherwise the system interpreter.
_python_bin() {
if [[ -x "$COURSE_ROOT/.venv/bin/python" ]]; then
printf '%s' "$COURSE_ROOT/.venv/bin/python"
else
printf 'python3'
fi
}
# Dependencies are the runner's job, not the reader's. Without this, EARIN
# dies on ModuleNotFoundError for gymnasium and the "runnable" status is a lie.
_python_install_requirements() {
local reqs=()
while IFS= read -r r; do reqs+=("$r"); done < <(
find "$COURSE_ROOT" -name 'requirements*.txt' -not -path '*/.venv/*' | sort)
[[ ${#reqs[@]} -gt 0 ]] || return 0
if [[ ! -x "$COURSE_ROOT/.venv/bin/python" ]]; then
printf 'creating .venv\n'
python3 -m venv "$COURSE_ROOT/.venv"
fi
"$COURSE_ROOT/.venv/bin/python" -m pip install --quiet --upgrade pip
# EVERY requirements file, not just the first: EARIN is six independent
# labs plus a project, and installing only one of them left the lab it
# then tried to run missing gymnasium.
local req
for req in "${reqs[@]}"; do
printf 'installing %s\n' "${req#"$COURSE_ROOT"/}"
"$COURSE_ROOT/.venv/bin/python" -m pip install --quiet -r "$req" || {
printf 'warning: some requirements in %s failed\n' "${req#"$COURSE_ROOT"/}" >&2
}
done
}
# Where a course ships real tests, running them is a stronger check than
# running the program -- and for an interactive program (ECRYPT_PROJECT prompts
# on stdin) it is the ONLY check that works without a person at a keyboard.
_python_run_tests() {
local suite
suite="$(find "$COURSE_ROOT" -name 'test_*.py' -not -path '*/.venv/*' -print -quit 2>/dev/null)"
[[ -n "$suite" ]] || return 0
local py
py="$(_python_bin)"
"$py" -c 'import pytest' 2> /dev/null || {
printf 'tests present but pytest is not installed; skipping\n'
return 0
}
# Which directory pytest runs FROM is course-specific and cannot be
# guessed: ECOTE needs program/ so `from translator.main import` resolves,
# ECRYPT_PROJECT needs sieve/ because its fixtures open P-100000.csv by a
# relative path. So the stub says, and COURSE_ROOT is only the default.
local from_dir="${COURSE_TEST_DIR:-$COURSE_ROOT}"
printf 'running the test suite in %s\n' "${from_dir#"$COURSE_ROOT"/}"
( cd "$from_dir" && "$py" -m pytest -q )
}
# --- runners -----------------------------------------------------------------
_run() {
case "$COURSE_KIND" in
none) _no_code "${COURSE_NOTE:-nothing to run}" ;;
tex)
local tex="${COURSE_ENTRY:-}"
[[ -n "$tex" ]] || tex="$(_first_file '*.tex')"
[[ -n "$tex" ]] || _no_code "no .tex found"
local tool
tool="$(_require_any 'a LaTeX toolchain' latexmk pdflatex xelatex)"
# Build into build/, never beside the source. A submitted report.pdf
# is a tracked deliverable, and a rebuild whose output name happens
# to match it would silently overwrite what was handed in.
local dir base
dir="$(dirname "$tex")"
base="$(basename "$tex")"
# A document that declares \bibliography but never cites anything
# makes bibtex exit 2 ("I found no \citation commands"), and
# latexmk turns that into exit 12 even though the PDF is fine.
# --no-bibtex says "this one was handed in without a bibliography",
# which is a fact about the deliverable, not a suppression.
# An `x && y=1` one-liner would be a set -e landmine here: when x
# is false the list returns 1 and the whole runner exits.
local bibflag=()
if [[ -n "${COURSE_TEX_NO_BIBTEX:-}" ]]; then
bibflag=(-bibtex-)
fi
if [[ $tool == latexmk ]]; then
( cd "$dir" && latexmk -pdf "${bibflag[@]}" \
-interaction=nonstopmode -outdir=build "$base" )
else
( cd "$dir" && mkdir -p build \
&& "$tool" -interaction=nonstopmode -output-directory=build "$base" )
fi
printf 'output in %s/build\n' "$dir"
;;
python)
local entry="${COURSE_ENTRY:-}"
# An entry may be given as a directory; resolve it to the module
# inside. Passing the directory itself makes python look for
# __main__.py and fail with a message about the wrong thing.
if [[ -d "$entry" ]]; then
local cand
for cand in main.py project.py __main__.py; do
if [[ -f "$entry/$cand" ]]; then
entry="$entry/$cand"
break
fi
done
fi
# Only auto-pick a main.py when there is exactly ONE. A course
# with six labs has six, and choosing whichever find returns first
# is not "running the project", it is running an arbitrary lab.
if [[ ! -f "$entry" ]]; then
local mains
mains="$(find "$COURSE_ROOT" -name 'main.py' -not -path '*/.venv/*' | wc -l)"
[[ "$mains" -eq 1 ]] && entry="$(_first_file 'main.py')"
fi
_python_install_requirements
if [[ -f "$entry" ]]; then
( cd "$(dirname "$entry")" && "$(_python_bin)" "$(basename "$entry")" )
else
# A pile of one-shot scripts with no entry contract (NLP is
# the case here). Syntax-check them all rather than picking one
# arbitrarily and calling that "running the project".
printf 'no single entry point; syntax-checking every module instead\n'
find "$COURSE_ROOT" -name '*.py' -not -path '*/.venv/*' -print0 \
| xargs -0 -r "$(_python_bin)" -m py_compile
printf 'all modules compile\n'
fi
_python_run_tests
;;
node) ( cd "${COURSE_ENTRY:-$COURSE_ROOT}" && npm install && npm start ) ;;
maven) ( cd "${COURSE_ENTRY:-$COURSE_ROOT}" && mvn -q package ) ;;
make) ( cd "${COURSE_ENTRY:-$COURSE_ROOT}" && make ) ;;
dotnet) ( cd "${COURSE_ENTRY:-$COURSE_ROOT}" && dotnet build ) ;;
compose) ( cd "${COURSE_ENTRY:-$COURSE_ROOT}" && docker compose up ) ;;
unity) _blocked "open ${COURSE_ROOT} in the Unity editor; there is no headless path" ;;
cxx)
# Each .cpp here is a self-contained assignment with its own main(),
# so they are compiled individually rather than linked together.
local cxx src rel out
cxx="$(_require_any 'a C++ compiler' g++ clang++)"
# Binaries go to build/, which is gitignored -- compiling in place
# drops untracked executables next to the sources, and an
# extensionless binary is not something .gitignore catches by name.
mkdir -p "$COURSE_ROOT/build"
while IFS= read -r src; do
rel="${src#"$COURSE_ROOT"/}"
if _is_known_incomplete "${src#"${COURSE_GATES%/gates}"/}"; then
printf 'skipping %s (see gates/known-incomplete.txt)\n' "$rel"
continue
fi
out="$COURSE_ROOT/build/$(basename "${rel%.cpp}")"
# Some files here are exam-answer fragments with no main(), so
# they cannot be linked. Compile those to an object instead --
# that still type-checks them, which is the point.
if grep -qE '\bint[[:space:]]+main[[:space:]]*\(' "$src"; then
printf 'compiling %s\n' "$rel"
"$cxx" -O2 -o "$out" "$src"
else
printf 'checking %s (no main; -c only)\n' "$rel"
"$cxx" -O2 -c -o "${out}.o" "$src"
fi
done < <(find "$COURSE_ROOT" -name '*.cpp' -not -path '*/node_modules/*' -not -path '*/build/*' | sort)
printf 'binaries in %s/build\n' "$COURSE_ROOT"
;;
notebook)
_have jupyter || _blocked "needs jupyter"
local nb
while IFS= read -r nb; do
jupyter nbconvert --to notebook --execute --inplace "$nb"
done < <(find "$COURSE_ROOT" -name '*.ipynb' -not -path '*/.ipynb_checkpoints/*' | sort)
;;
matlab)
local tool
tool="$(_require_any 'MATLAB or Octave' matlab octave)"
printf 'run the .m files in %s with %s\n' "$COURSE_ROOT" "$tool"
;;
*) printf 'unknown COURSE_KIND: %s\n' "${COURSE_KIND:-unset}" >&2; return 1 ;;
esac
}
course_main() { course_main() {
COURSE_KIND="" COURSE_KIND=""
@ -333,6 +100,7 @@ course_main() {
COURSE_NOTE="" COURSE_NOTE=""
COURSE_TEST_DIR="" COURSE_TEST_DIR=""
COURSE_TEX_NO_BIBTEX="" COURSE_TEX_NO_BIBTEX=""
COURSE_ENTRY_ARGS=""
while [[ $# -gt 0 ]]; do while [[ $# -gt 0 ]]; do
case "$1" in case "$1" in
--kind) COURSE_KIND="$2"; shift 2 ;; --kind) COURSE_KIND="$2"; shift 2 ;;
@ -340,6 +108,7 @@ course_main() {
--note) COURSE_NOTE="$2"; shift 2 ;; --note) COURSE_NOTE="$2"; shift 2 ;;
--test-dir) COURSE_TEST_DIR="$2"; shift 2 ;; --test-dir) COURSE_TEST_DIR="$2"; shift 2 ;;
--no-bibtex) COURSE_TEX_NO_BIBTEX=1; shift ;; --no-bibtex) COURSE_TEX_NO_BIBTEX=1; shift ;;
--entry-args) COURSE_ENTRY_ARGS="$2"; shift 2 ;;
--) shift; break ;; --) shift; break ;;
*) break ;; *) break ;;
esac esac

70
scripts/course_probe.sh Normal file
View File

@ -0,0 +1,70 @@
# shellcheck shell=bash
# Split out of course_lib.sh, which was over this repo's 250-line cap. Sourced
# by it, never on its own -- the helpers and constants it uses live there.
#
# Answers --probe: can this course run HERE, without running it.
# --- probes ------------------------------------------------------------------
_probe() {
case "$COURSE_KIND" in
none) _no_code "${COURSE_NOTE:-no executable code in this directory}" ;;
tex)
_probe_any "a LaTeX toolchain" latexmk pdflatex xelatex
_runnable "builds ${COURSE_ENTRY:-the report} with latexmk"
;;
python)
_have python3 || _blocked "needs python3"
[[ -n "$(_first_file '*.py')" ]] || _no_code "no .py files found"
_runnable "python3 ${COURSE_ENTRY:-the entry script}"
;;
node)
_have node || _blocked "needs node"
_have npm || _blocked "needs npm"
_runnable "npm install && npm start"
;;
maven)
_have mvn || _blocked "needs maven"
_have java || _blocked "needs a JDK"
_runnable "mvn package"
;;
make)
_have make || _blocked "needs make"
_probe_any "a C compiler" gcc cc clang
_runnable "make"
;;
dotnet)
_have dotnet || _blocked "needs the .NET SDK"
_runnable "dotnet build"
;;
unity)
_have unity-editor || _have unityhub \
|| _blocked "needs the Unity editor (GUI, not installable on a CI runner)"
_runnable "opens in the Unity editor"
;;
matlab)
_probe_any "MATLAB or Octave" matlab octave
_runnable "runs the .m scripts"
;;
cxx)
_probe_any "a C++ compiler" g++ clang++
[[ -n "$(_first_file '*.cpp')" ]] || _no_code "no .cpp files found"
_runnable "compiles each standalone .cpp"
;;
notebook)
_have jupyter || _blocked "needs jupyter"
# Being on PATH is not the same as working: this machine has a
# jupyter shim whose pipx venv interpreter is gone, so it fails
# with "bad interpreter". Probe by actually invoking it.
jupyter --version > /dev/null 2>&1 || _blocked "jupyter is on PATH but broken (bad interpreter)"
_runnable "jupyter nbconvert --execute"
;;
compose)
_have docker || _blocked "needs docker"
docker info > /dev/null 2>&1 || _blocked "docker is installed but the daemon is not running"
_runnable "docker compose up"
;;
*) _blocked "unknown COURSE_KIND '${COURSE_KIND:-unset}'" ;;
esac
}

198
scripts/course_run.sh Normal file
View File

@ -0,0 +1,198 @@
# shellcheck shell=bash
# Split out of course_lib.sh, which was over this repo's 250-line cap. Sourced
# by it, never on its own -- the helpers and constants it uses live there.
#
# Answers a bare invocation: actually build or run the course.
# Use the course venv when one exists, otherwise the system interpreter.
_python_bin() {
if [[ -x "$COURSE_ROOT/.venv/bin/python" ]]; then
printf '%s' "$COURSE_ROOT/.venv/bin/python"
else
printf 'python3'
fi
}
# Dependencies are the runner's job, not the reader's. Without this, EARIN
# dies on ModuleNotFoundError for gymnasium and the "runnable" status is a lie.
_python_install_requirements() {
local reqs=()
while IFS= read -r r; do reqs+=("$r"); done < <(
find "$COURSE_ROOT" -name 'requirements*.txt' -not -path '*/.venv/*' | sort)
[[ ${#reqs[@]} -gt 0 ]] || return 0
if [[ ! -x "$COURSE_ROOT/.venv/bin/python" ]]; then
printf 'creating .venv\n'
python3 -m venv "$COURSE_ROOT/.venv"
fi
"$COURSE_ROOT/.venv/bin/python" -m pip install --quiet --upgrade pip
# EVERY requirements file, not just the first: EARIN is six independent
# labs plus a project, and installing only one of them left the lab it
# then tried to run missing gymnasium.
local req
for req in "${reqs[@]}"; do
printf 'installing %s\n' "${req#"$COURSE_ROOT"/}"
"$COURSE_ROOT/.venv/bin/python" -m pip install --quiet -r "$req" || {
printf 'warning: some requirements in %s failed\n' "${req#"$COURSE_ROOT"/}" >&2
}
done
}
# Where a course ships real tests, running them is a stronger check than
# running the program -- and for an interactive program (ECRYPT_PROJECT prompts
# on stdin) it is the ONLY check that works without a person at a keyboard.
_python_run_tests() {
local suite
suite="$(find "$COURSE_ROOT" -name 'test_*.py' -not -path '*/.venv/*' -print -quit 2>/dev/null)"
[[ -n "$suite" ]] || return 0
local py
py="$(_python_bin)"
"$py" -c 'import pytest' 2> /dev/null || {
printf 'tests present but pytest is not installed; skipping\n'
return 0
}
# Which directory pytest runs FROM is course-specific and cannot be
# guessed: ECOTE needs program/ so `from translator.main import` resolves,
# ECRYPT_PROJECT needs sieve/ because its fixtures open P-100000.csv by a
# relative path. So the stub says, and COURSE_ROOT is only the default.
local from_dir="${COURSE_TEST_DIR:-$COURSE_ROOT}"
printf 'running the test suite in %s\n' "${from_dir#"$COURSE_ROOT"/}"
( cd "$from_dir" && "$py" -m pytest -q )
}
# --- runners -----------------------------------------------------------------
_run() {
case "$COURSE_KIND" in
none) _no_code "${COURSE_NOTE:-nothing to run}" ;;
tex)
local tex="${COURSE_ENTRY:-}"
[[ -n "$tex" ]] || tex="$(_first_file '*.tex')"
[[ -n "$tex" ]] || _no_code "no .tex found"
local tool
tool="$(_require_any 'a LaTeX toolchain' latexmk pdflatex xelatex)"
# Build into build/, never beside the source. A submitted report.pdf
# is a tracked deliverable, and a rebuild whose output name happens
# to match it would silently overwrite what was handed in.
local dir base
dir="$(dirname "$tex")"
base="$(basename "$tex")"
# A document that declares \bibliography but never cites anything
# makes bibtex exit 2 ("I found no \citation commands"), and
# latexmk turns that into exit 12 even though the PDF is fine.
# --no-bibtex says "this one was handed in without a bibliography",
# which is a fact about the deliverable, not a suppression.
# An `x && y=1` one-liner would be a set -e landmine here: when x
# is false the list returns 1 and the whole runner exits.
local bibflag=()
if [[ -n "${COURSE_TEX_NO_BIBTEX:-}" ]]; then
bibflag=(-bibtex-)
fi
if [[ $tool == latexmk ]]; then
( cd "$dir" && latexmk -pdf "${bibflag[@]}" \
-interaction=nonstopmode -outdir=build "$base" )
else
( cd "$dir" && mkdir -p build \
&& "$tool" -interaction=nonstopmode -output-directory=build "$base" )
fi
printf 'output in %s/build\n' "$dir"
;;
python)
local entry="${COURSE_ENTRY:-}"
# An entry may be given as a directory; resolve it to the module
# inside. Passing the directory itself makes python look for
# __main__.py and fail with a message about the wrong thing.
if [[ -d "$entry" ]]; then
local cand
for cand in main.py project.py __main__.py; do
if [[ -f "$entry/$cand" ]]; then
entry="$entry/$cand"
break
fi
done
fi
# Only auto-pick a main.py when there is exactly ONE. A course
# with six labs has six, and choosing whichever find returns first
# is not "running the project", it is running an arbitrary lab.
if [[ ! -f "$entry" ]]; then
local mains
mains="$(find "$COURSE_ROOT" -name 'main.py' -not -path '*/.venv/*' | wc -l)"
[[ "$mains" -eq 1 ]] && entry="$(_first_file 'main.py')"
fi
_python_install_requirements
if [[ -f "$entry" ]]; then
# Word-split on purpose: --entry-args is a fixed string the
# course stub writes, never user input, and it has to become
# several argv entries. read -a keeps that explicit.
local entry_args=()
if [[ -n "${COURSE_ENTRY_ARGS:-}" ]]; then
read -r -a entry_args <<< "$COURSE_ENTRY_ARGS"
fi
( cd "$(dirname "$entry")" \
&& "$(_python_bin)" "$(basename "$entry")" "${entry_args[@]}" )
else
# A pile of one-shot scripts with no entry contract (NLP is
# the case here). Syntax-check them all rather than picking one
# arbitrarily and calling that "running the project".
printf 'no single entry point; syntax-checking every module instead\n'
find "$COURSE_ROOT" -name '*.py' -not -path '*/.venv/*' -print0 \
| xargs -0 -r "$(_python_bin)" -m py_compile
printf 'all modules compile\n'
fi
_python_run_tests
;;
node) ( cd "${COURSE_ENTRY:-$COURSE_ROOT}" && npm install && npm start ) ;;
maven) ( cd "${COURSE_ENTRY:-$COURSE_ROOT}" && mvn -q package ) ;;
make) ( cd "${COURSE_ENTRY:-$COURSE_ROOT}" && make ) ;;
dotnet) ( cd "${COURSE_ENTRY:-$COURSE_ROOT}" && dotnet build ) ;;
compose) ( cd "${COURSE_ENTRY:-$COURSE_ROOT}" && docker compose up ) ;;
unity) _blocked "open ${COURSE_ROOT} in the Unity editor; there is no headless path" ;;
cxx)
# Each .cpp here is a self-contained assignment with its own main(),
# so they are compiled individually rather than linked together.
local cxx src rel out
cxx="$(_require_any 'a C++ compiler' g++ clang++)"
# Binaries go to build/, which is gitignored -- compiling in place
# drops untracked executables next to the sources, and an
# extensionless binary is not something .gitignore catches by name.
mkdir -p "$COURSE_ROOT/build"
while IFS= read -r src; do
rel="${src#"$COURSE_ROOT"/}"
if _is_known_incomplete "${src#"${COURSE_GATES%/gates}"/}"; then
printf 'skipping %s (see gates/known-incomplete.txt)\n' "$rel"
continue
fi
out="$COURSE_ROOT/build/$(basename "${rel%.cpp}")"
# Some files here are exam-answer fragments with no main(), so
# they cannot be linked. Compile those to an object instead --
# that still type-checks them, which is the point.
if grep -qE '\bint[[:space:]]+main[[:space:]]*\(' "$src"; then
printf 'compiling %s\n' "$rel"
"$cxx" -O2 -o "$out" "$src"
else
printf 'checking %s (no main; -c only)\n' "$rel"
"$cxx" -O2 -c -o "${out}.o" "$src"
fi
done < <(find "$COURSE_ROOT" -name '*.cpp' -not -path '*/node_modules/*' -not -path '*/build/*' | sort)
printf 'binaries in %s/build\n' "$COURSE_ROOT"
;;
notebook)
_have jupyter || _blocked "needs jupyter"
local nb
while IFS= read -r nb; do
jupyter nbconvert --to notebook --execute --inplace "$nb"
done < <(find "$COURSE_ROOT" -name '*.ipynb' -not -path '*/.ipynb_checkpoints/*' | sort)
;;
matlab)
local tool
tool="$(_require_any 'MATLAB or Octave' matlab octave)"
printf 'run the .m files in %s with %s\n' "$COURSE_ROOT" "$tool"
;;
*) printf 'unknown COURSE_KIND: %s\n' "${COURSE_KIND:-unset}" >&2; return 1 ;;
esac
}