Skip to content
Merged
23 changes: 19 additions & 4 deletions .github/unraid-watch/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,14 @@ plugin stands. Both go to issue #52.

- **check** (every CI run, offline). Reads each GraphQL document out of `scripts/*.js` and `*.page`, and each
webgui PHP file the plugin references. Fails when a document is not a plain string, because nothing
downstream could see it.
downstream could see it. Also fails on a `var(--x)` in the plugin's stylesheets with no fallback when only
Unraid defines `--x`, so an Unraid release without it cannot drop the style; a variable the plugin's own
stylesheets declare, or its scripts or server code name (the CSS tool's presets), is the plugin's.
- **contract** (pull requests that touch `src/`, this directory or the workflow file, every push to `main` or
`beta`, and daily for both). Validates every operation against the `unraid/api` schema on `main` and at the
latest release, and confirms every watched path still exists upstream. Red means the plugin sends something
that is no longer valid, or a file it depends on is gone. The daily run checks each branch with that branch's
latest release, confirms every watched path still exists upstream, and checks every contract on each stable
webgui release tag from the plugin's declared minimum and on the development branch. Red means the plugin sends
something that is no longer valid, a file it depends on is gone, or a release it supports breaks a contract. The daily run checks each branch with that branch's
copy of the tool; a manual run checks the branch it is started on.
- **watch** (daily, on every push to `beta`, and by hand). It always reads the plugin on `beta`, whichever branch
starts it, and does two things:
Expand Down Expand Up @@ -39,8 +42,14 @@ Once it is on `main` the daily run starts by itself, and still reads `beta`.
- **Coming with X.Y**: what the newest OS pre-release has that the newest stable does not.
- **Operations by OS release**: how many of the plugin's operations are valid on the API each OS release
bundles, and which fields are missing.
- **Contracts by OS release**: whether each contract holds on every webgui release tag from the plugin's minimum
and on the development branch. An OS release the webgui has no tag for (some patch releases) is not read.
- **Unraid CSS by OS release**: per theme, the layout properties Unraid's stylesheets set on an element the
plugin's own CSS styles, that differ between releases, and that the plugin does not set itself: what can
look different on an older or newer Unraid. A rule no longer set on the development branch is listed apart,
as it has often only moved to a stylesheet not read here.
- **API not used yet**: Docker and VM operations the plugin never calls, and how much of each type it reads.
- **Still on legacy**: the webgui files the plugin still depends on.
- **Still on legacy**: the webgui files the plugin still depends on, stylesheets aside.

OS versions and the API each one bundles are read from the release notes. They state it in prose, in several
styles; a release whose notes do not say is listed as not stated rather than guessed. Files are compared by
Expand All @@ -63,6 +72,12 @@ with `' + name + '` when `watch.json` has an `expand` rule naming the JS map tha
operation from an unread field.
- `releases`: directory holding the OS release notes. `since` is the oldest OS release to list, and `tags` is
the repository whose tags carry those OS versions.
- `manifest`: the plugin's `.plg`. Its `min` attribute is the oldest OS release the contracts and stylesheets
are checked on; the webgui's stable tags from there up are the releases, and finding none is an error.
- `contracts`: what the plugin relies on in a webgui file, as a `path`, a regular expression it must `match`,
and `why`. Optional `since`: the oldest release held to it. Add one with each change that relies on Unraid.
- `styles`: the `themes`, and the stylesheet `files` that make up each one in load order (`{theme}` is
replaced). A file a release does not have is skipped, so files from older and newer layouts can both be listed.

## Running it by hand

Expand Down
185 changes: 185 additions & 0 deletions .github/unraid-watch/css.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,185 @@
"""Stylesheets, FolderView3's and Unraid's: rules as text, the elements FolderView3 styles, and what Unraid sets on them per release."""
import re

TOKEN = re.compile(r'[#.][A-Za-z_][\w-]*')
COMMENT = re.compile(r'/\*.*?\*/', re.S)
VAR = re.compile(r'var\(\s*(--[\w-]+)\s*([,)])')
DEFINED = re.compile(r'(?:^|[;{\s])(--[\w-]+)\s*:')
# At-rules whose block holds rules for elements; @keyframes, @font-face and @page hold none
GROUPS = ('@media', '@supports', '@container', '@layer', '@scope')
# What moves or wraps an element. Colours and fonts change with each theme by design, so they are left out
LAYOUT = frozenset({
'display', 'position', 'white-space', 'overflow', 'overflow-x', 'overflow-y', 'float', 'clear', 'box-sizing', 'flex', 'flex-direction',
'flex-wrap', 'flex-grow', 'flex-shrink', 'flex-basis', 'align-items', 'justify-content', 'gap', 'grid-template-columns', 'width',
'min-width', 'max-width', 'height', 'min-height', 'max-height', 'margin', 'margin-top', 'margin-bottom', 'margin-left', 'margin-right',
'padding', 'padding-top', 'padding-bottom', 'padding-left', 'padding-right', 'line-height', 'font-size', 'vertical-align', 'text-align',
'text-overflow', 'word-break', 'overflow-wrap', 'table-layout', 'border-collapse', 'border-spacing', 'top', 'left', 'right', 'bottom',
'transform', 'z-index', 'visibility', 'content'})


def uncommented(text):
"""The text with each comment blanked out, line breaks kept so positions still give line numbers."""
return COMMENT.sub(lambda m: '\n' * m.group(0).count('\n'), text)


def _plain(text):
"""[(index, char)] for characters outside quoted strings, so a "}" or ";" in a content value is just text."""
out, quote = [], None
for i, c in enumerate(text):
if quote:
quote = None if c == quote and text[i - 1] != '\\' else quote
elif c in '"\'':
quote = c
else:
out.append((i, c))
return out


def _split(text, at):
"""text cut at each `at` character outside parentheses and strings (a url() or :not() list keeps its ; and ,)."""
parts, depth, start = [], 0, 0
for i, c in _plain(text):
depth += (c == '(') - (c == ')')
if c == at and depth <= 0:
parts.append(text[start:i])
start = i + 1
return parts + [text[start:]]


def _selector(text):
"""One spelling per selector: spacing around combinators and quotes around attribute values do not make a rule new."""
text = re.sub(r'\s*([>+~])\s*', r' \1 ', text)
text = re.sub(r'\[\s*([\w-]+)\s*([~|^$*]?=)\s*["\']?([^"\'\]]*?)["\']?\s*\]', r'[\1\2\3]', text)
return ' '.join(text.split())


def _condition(text):
"""One spelling per at-rule condition, `@media (max-width: 767px)`, however the stylesheet spaced it."""
text = re.sub(r'\s*([():,])\s*', r'\1', text)
text = re.sub(r'(?<=[\w)])\(', ' (', text).replace(':', ': ').replace(',', ', ')
return ' '.join(text.split())


def _parts(body):
"""[(None, declarations) or (head, block)] of a body in source order: its own declarations and the rules nested in it."""
items, depth, paren, start, item = [], 0, 0, 0, 0
for i, c in _plain(body):
paren += (c == '(') - (c == ')') if depth == 0 else 0
if c == '{':
if depth == 0:
items.append((None, body[start:item]))
head, opened = ' '.join(body[item:i].split()), i + 1
depth += 1
elif c == '}' and depth:
depth -= 1
if depth == 0:
items.append((head, body[opened:i]))
start = item = i + 1
elif c == ';' and depth == 0 and paren <= 0:
item = i + 1
return items + [(None, body[start:])]


def _set(decls, name, value):
"""A later declaration wins, except over an !important one that it is not itself."""
if not (decls.get(name, '').endswith('!important') and not value.endswith('!important')):
decls[name] = value


def _declarations(text):
decls = {}
for part in _split(text, ';'):
name, colon, value = part.partition(':')
if colon and name.strip() and not name.strip().startswith('@'):
# `4px!important` and `4px !important` are the same value
_set(decls, name.strip().lower(), ' '.join(re.sub(r'\s*!\s*important\b', ' !important', value).split()))
return decls


def _walk(body, conditions, parents, out):
# In source order: a declaration after a nested rule comes after it, as the browser applies them
for head, block in _parts(body):
if head is None:
decls = _declarations(block) if parents else {}
out += [(conditions, parent, decls) for parent in parents if decls]
continue
if head.startswith('@'):
# `@media(max-width:600px)` needs no space after the keyword
if re.match(r'@[\w-]+', head).group(0).lower() in GROUPS:
_walk(block, ' '.join(filter(None, (conditions, _condition(head)))), parents, out)
continue
selectors = [_selector(s) for s in _split(head, ',') if s.strip()]
# Nesting: & is the parent rule; a nested selector without one is a descendant of it
resolved = [s.replace('&', p) if '&' in s else f'{p} {s}' for p in parents for s in selectors] if parents else selectors
_walk(block, conditions, resolved, out)


def rules(text):
"""[(at-rule conditions, selector, {property: value})] in document order, nested rules resolved."""
out = []
_walk(uncommented(text), '', [], out)
return out


def subject(selector):
"""The ids and classes of the element a selector styles; none for a ::before box, nothing an attribute value or :not() names."""
# Innermost first, until none is left: 7.4 wraps its guards as :where(:not(.unapi *))
bare, previous = re.sub(r'\[[^\]]*\]', '', selector), None
while bare != previous:
bare, previous = re.sub(r'\([^()]*\)', '', bare), bare
Comment thread
chodeus marked this conversation as resolved.
Resolved
last = re.split(r'[\s>+~]+', bare.strip())[-1]
return [] if '::' in last else TOKEN.findall(last)


def styled(texts):
"""{id or class: {(at-rule conditions, property)} FolderView3 sets on elements that carry it} from its own stylesheets."""
out = {}
for text in texts:
for conditions, selector, decls in rules(text):
for token in subject(selector):
out.setdefault(token, set()).update((conditions, name) for name in decls)
return out


def merged(texts):
"""{(at-rule conditions, selector, property): value} for stylesheets in load order: a later one wins, !important aside."""
out = {}
for text in texts:
for conditions, selector, decls in rules(text):
for name, value in decls.items():
_set(out, (conditions, selector, name), value)
return out


def _covers(ours, token, conditions, name):
# The plugin's own rule for this property covers Unraid's when it holds everywhere, or under the same condition
return ('', name) in ours[token] or (conditions, name) in ours[token]


def changes(by_ref, refs, ours):
"""[(conditions, selector, property, [(ref, value)])]: layout Unraid changed on elements FolderView3 styles but does not set."""
out = []
for conditions, selector, name in sorted(set().union(*(by_ref[r] for r in refs))):
tokens = [t for t in subject(selector) if t in ours]
if name not in LAYOUT or not tokens or any(_covers(ours, t, conditions, name) for t in tokens):
continue
values = [(ref, by_ref[ref].get((conditions, selector, name))) for ref in refs]
if len({v for _, v in values}) > 1:
out.append((conditions, selector, name, values))
return out


def fallbacks(stylesheets, code):
"""Problems: var(--x) with no fallback where --x is Unraid's, not declared or keyed by FolderView3's own CSS or code."""
texts = {f: uncommented(f.read_text(encoding='utf-8')) for f in stylesheets}
ours = {name for text in texts.values() for name in DEFINED.findall(text)}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
named = '\n'.join(f.read_text(encoding='utf-8') for f in code)

def keyed(name):
# The CSS tool's presets ('fv3-x': '#fff') and the generated theme CSS ('x' => '#fff') key the plugin's own variables;
# a name that is only read (getPropertyValue('--x')) is not defined by that
return re.search(r"['\"](?:--)?" + re.escape(name[2:]) + r"['\"]\s*(?::|=>)", named)
return [f'{f.name}:{text.count(chr(10), 0, m.start()) + 1}: var({m.group(1)}) has no fallback and FolderView3 does not '
f'define {m.group(1)}, so the style needs every Unraid release to; give it a fallback'
for f, text in texts.items() for m in VAR.finditer(text)
if m.group(2) == ')' and m.group(1) not in ours and not keyed(m.group(1))]
16 changes: 16 additions & 0 deletions .github/unraid-watch/github.py
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,22 @@ def read():
return self._call('GET', f'/repos/{repo}/contents/{urllib.parse.quote(path)}', {'ref': ref}, raw=True)
return self._once(('file', repo, ref, path), read)

def blob(self, repo, sha):
"""A file's text by its blob sha: one read per version of a file, however many releases share it."""
return self._once(('blob', repo, sha), lambda: self._call('GET', f'/repos/{repo}/git/blobs/{sha}', raw=True))

def tags(self, repo):
"""Every tag name of a repository."""
def read():
names = []
for page in range(1, 51):
batch = self._call('GET', f'/repos/{repo}/tags', {'page': page, 'per_page': 100})
names += [t['name'] for t in batch]
if len(batch) < 100:
return names
raise GitHubError(f'{repo}: more than 5000 tags')
return self._once(('tags', repo), read)

def exists(self, repo, ref, path):
return bool(self._call('HEAD', f'/repos/{repo}/contents/{urllib.parse.quote(path.rstrip("/"))}',
{'ref': ref}, missing=(404,)))
Expand Down
6 changes: 6 additions & 0 deletions .github/unraid-watch/releases.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@ def stable(version):
return '-' not in version


def os_tag(name):
"""A stable Unraid OS release tag such as 7.4.0; the webgui's oldest tags are dates such as 2014.11.10."""
found = VERSION.fullmatch(name)
return bool(found) and stable(name) and int(found.group(1)) < 1000


def heading(text):
"""First markdown heading of a release-notes file; '' when it has none."""
for line in text.splitlines():
Expand Down
55 changes: 55 additions & 0 deletions .github/unraid-watch/report.py
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,59 @@ def _changed(commits):
return f'{len(commits)} commit{"" if len(commits) == 1 else "s"}, newest {code(f"{date} {subject}")}'


def _runs(values):
"""`a` in `7.0.0`–`7.3.5`, not set in `7.4.0`–`master`: consecutive refs with the same value as one span."""
runs = []
for ref, value in values:
if runs and runs[-1][0] == value:
runs[-1][2] = ref
else:
runs.append([value, ref, ref])
return ', '.join(f'{"not set" if value is None else code(value)} in {code(a)}' + ('' if a == b else f'–{code(b)}')
for value, a, b in runs)


def _contracts(view):
if 'unread' in view:
return ['', f'**Contracts and Unraid CSS by OS release**: could not be read ({code(view["unread"], 300)}).']
*tags, branch = view['refs']
span = f'{code(tags[0])} to {code(tags[-1])}' if len(tags) > 1 else code(tags[0])
out = ['', '**Contracts by OS release**', '',
f'What FolderView3 relies on in the webgui, checked on its release tags {span} and on {code(branch)}.', '']
for entry, found in view['contracts']:
held = [ref for ref, ok in found.items() if ok is not None]
broken = [f'{"missing" if ok == "missing" else "fails"} on {code(ref)}' for ref, ok in found.items() if ok not in (True, None)]
since = f' (from {code(entry["since"])})' if 'since' in entry else ''
out.append(f'- {code(entry["path"].rsplit("/", 1)[-1])} {entry["why"]}: '
+ (', '.join(broken) if broken else f'holds on all {len(held)}') + since)
return out


def _styles(view):
if 'unread' in view:
return []
themes, changed, gone = list(view['styles']), {}, {}
for theme, rows in view['styles'].items():
for context, selector, name, values in rows:
# Not set on the development branch any more: often a rule that moved to a stylesheet this does not read
(gone if values[-1][1] is None else changed).setdefault((context, selector, name, tuple(values)), []).append(theme)

def row(key, who):
context, selector, name, values = key
where = '' if len(who) == len(themes) else f' ({", ".join(who)})'
return f'- {code((context + " " if context else "") + selector, 200)} {code(name)}: {_runs(values)}{where}'
out = ['', '**Unraid CSS by OS release**', '',
'Layout properties Unraid sets on elements FolderView3 styles, that differ between releases, and that FolderView3 '
'does not set itself.', '']
out += capped([row(k, w) for k, w in changed.items()], 40) or ['- none']
if view['unmatched']:
out += ['', 'Listed in watch.json but on no release: ' + ', '.join(code(p, 200) for p in view['unmatched']) + '.']
if gone:
out += ['', f'<details><summary>No longer set in the stylesheets read here ({len(gone)}; may have moved elsewhere)</summary>', '']
out += capped([row(k, w) for k, w in gone.items()], 40) + ['', '</details>']
return out


def render_status(view):
newest, stable, latest = view['releases']
facts = [f'Newest OS release {code(newest[0])} (API {code(newest[1] or "not stated")})'] if newest else []
Expand Down Expand Up @@ -124,6 +177,8 @@ def render_status(view):
+ ', '.join(f'{code(name)} {read}/{total}' for name, read, total in view['types']) + '.']
out += ['', '</details>', '', f'<details><summary>Still on legacy ({len(view["legacy"])} webgui files)</summary>', '']
out += [f'- {code(path, 200)}: {why}' for path, why in view['legacy']] + ['', '</details>']
# Last: the longest sections, so a body cut for length loses these before anything above
out += _contracts(view) + _styles(view)
return fit('\n'.join(out)) + '\n\n' + STATUS


Expand Down
15 changes: 15 additions & 0 deletions .github/unraid-watch/scan.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,21 @@ def operations(plugin, rules):
return ops, problems


def stylesheets(plugin):
return _files(plugin, ('styles/*.css',))


def code(plugin):
"""The scripts and server code, which name the plugin's own CSS variables (the CSS tool, the generated theme CSS)."""
return _files(plugin, ('scripts/*.js', 'server/*.php'))


def minimum(manifest):
"""The oldest Unraid release the plugin declares it runs on: the PLUGIN tag's min attribute; None when it has none."""
found = re.search(r'<PLUGIN\b[^>]*\bmin="(\d+\.\d+\.\d+)"', manifest.read_text(encoding='utf-8'))
return found and found.group(1)


def endpoints(plugin, prefixes):
"""{upstream path: file:line} for every webgui PHP file the plugin references by URL or include."""
found = {}
Expand Down
Loading
Loading