Elk Development => Feature Discussion => Exterminated Features => Topic started by: TestMonkey on December 20, 2012, 05:43:48 pm
Title: Code reworking: directory structure
Post by: TestMonkey on December 20, 2012, 05:43:48 pm
I need your input on this.
Issue: Currently we have /Sources, with 133 files (really), and a subdirectory Sources/lib with 2 external dependencies. While /Themes/default/ has 52 template files, and subdirectories for images, scripts, etc. The 133 files in /Sources are of several kinds:
controllers (admin and user actions handlers, i.e. ManageXxx and Profile.php etc) (about 70 files)
Subs- files (utility, library files) (31 files)
Class- files (6 defined at this time. We will add more, but I expect a maximum of 10-12 for version 1.0 or something)
other core files (database, about 9 files without those in Subs-, search classes (I didn't count them above cuz of name), core files like Load.php, Subs.php, Security.php etc.)
I need to reorganize these files, we are removing a few but we are adding more than we remove (at least twice more), as we rework the code... There are more options to do so... Which is more understandable, or your preference, or you see better reasons to, easier to work with?
Option 1: use a /Sources/controllers directory.
/Sources/controllers will hold controllers (in time they'd be Admin.controller.php), (about 70 files)
Themes/default/ holds templates (our known xxx.template.php), roughly a template file [should] correspond to a controller file (52 files)
/Sources/lib = external dependencies (as it is already)
/Sources = the rest, library [Subs-xxx], library classes [Class-xxx], etc. (we're left with about 60+ files)
Please see and browse :) https://github.com/norv/Dialogue/tree/dirs_test1
Sub-option... only admin controllers separated from the rest, i.e. https://github.com/norv/Dialogue/commit/f19ee9d277534d8ca3ca1b93cd75ab2718bc31f2 Sub-option... only admin, but all related to it, separated as /admin... (controllers + Class-xxx utility + Subs-xxx utility, all admin stuff)
the rest, basically the action handlers, what I called controller files, remain in /Sources (aprox. 70 files)
Problem with this, IMHO, is that the external libraries, the 2 we have in /lib currently, are no longer clearly separated. This isn't a good idea, the best practice (and expected by developers and automated build tools alike) is for them to be clearly in some easily distinguishable directory. Many name it /vendor. Sub-option: move the external libraries to /Sources/vendor. https://github.com/norv/Dialogue/tree/dirs_test2
Constraint of all options... We have a $sourcedir refered many many times in the files, which is used by people to secure the files that don't need to be in the public web directory. I think it would be easier if at this time, $sourcedir remains exactly that: a single variable to set to another location, outside "public_html" (whatever it's named), where people may move their entire /Sources dir and its subdirs. So I wouldn't "take out" of /Sources any of these dirs ("controllers" or "lib").
Which is better and why?
Title: Re: Code reworking: directory structure
Post by: Antechinus on December 20, 2012, 06:52:15 pm
Option 3: Reduce the number of files. Really.
Title: Re: Code reworking: directory structure
Post by: TestMonkey on December 20, 2012, 07:09:49 pm
I really don't think that will happen soon (at all). We do where appropriate, but it cannot be done just like that: on the contrary, 4000-lines files is insane. (usually; and those doing everything under the sun and don't forget their mother's shoes, moreso.)
1/ I'll note that the redesign we're doing already removes code, because it removes duplicate code. (by consolidating the duplicate common stuff in library functions). Sometimes it does arrive to the point of removing useless files, yes, but more often than not, it doesn't. (not now, for a 1.0+ version).
2/ The longer-term goal is to redesign the code such that features/functionality become more modular. Which answers the "reduce the number of files" differently: the admin can always remove (or not download and install at all) the modules.
ETA, 3/ Well hey, we cannot have new features without adding new code. 1/ and 2/ exist exactly to address that. Make it possible, with better designed code.
Title: Re: Code reworking: directory structure
Post by: Spuds on December 21, 2012, 09:59:45 am
Yeah ... I don't think we are done adding TBH ... but it will be easier to find where a function is located. Plus feature cat wants to add more :D
I like the idea of the controllers in Sources and then the helpers in Sources/helpers (pick your name) and leave Sources/lib for the things that are not ours ... library functions that we make use of .... Or go even further and bring helps up to the same level as Sources.
The Monkey and I were talking about this a bit, and no matter what we do, we only want to do it once (the dir structure) and then we needs to live with it, so lets make sure we have a structure that makes the most sense going forward
Title: Re: Code reworking: directory structure
Post by: emanuele on December 21, 2012, 10:59:07 am
Quote from: Spuds – The Monkey and I were talking about this a bit, and no matter what we do, we only want to do it once (the dir structure) and then we needs to live with it, so lets make sure we have a structure that makes the most sense going forward
I don't believe you. :P
Title: Re: Code reworking: directory structure
Post by: Spuds on December 21, 2012, 12:04:54 pm
LOL ... you can believe that I at least said it :P
Title: Re: Code reworking: directory structure
Post by: TestMonkey on December 21, 2012, 02:36:10 pm
A quick update for you folks to browse around. Sorry for being on the run at the moment. I will come back on it to eventually discuss some of the aspects involved here. Short story is... Drivers for the solution are: the next two major versions, addons expectations and good practices (please take a look in /Packages/gallery too), and still within constraints (like mentioned in the OP) and in a way, a maximum of compatibility with our current codebase and habits of work... (well where this maximum is lower than the first two concerns :D)
https://github.com/norv/Dialogue/tree/dirs_test4
Title: Re: Code reworking: directory structure
Post by: TestMonkey on December 21, 2012, 05:03:46 pm
OK :D... Ema said moar...
Split some modules of the main body: first candidates are imo those with many actions/subactions (those with a menu of their own, admin, mod, profile and pm), and auth, search, attachment, calendar and packages. (I 'made' the first several) https://github.com/norv/Dialogue/tree/dirs_test5
Please see /Sources... respectively /database and /ext are there too. Oh, and 'the rest' are in /forum... that is, forum module.
What will happen in this case is that many *.subs.php files will be cross-used imho. Which is fine, but then we need a way to locate them immediately, to type easily. Such as, function loadLib ($modulename, $filename = ''), i.e. loadLib ('pm'); will load all .subs and .class in module 'pm' (there's only one :D), or loadLib ('pm', 'PersonalMessage') will load specifically this .subs file.
There is no claim (other than naming patterns and action handling from dispatcher; respectively a few functions like loadModule() as Ema wants or loadLib()) that these modules are anything 'more' than what they were: only functionality is separated in their own subdirs. That is, there's no question of managing them, install/uninstall and so on.
Title: Re: Code reworking: directory structure
Post by: Spuds on December 21, 2012, 05:30:08 pm
Having the various levels to look at really helps, thanks for doing that.
To me the 4th is about as far as I would go, the 5th really seems excessive to me ... at that point you would begin to wonder where to put things, at least I would. And no no no need to explain it for me :P .... its just an IMO.
I'll cruse around a bit more on it ... but just seems when you get as granular as the 5th .... its just to compartmentalized and therefore subject to breaking where we need YAD (yet another directory) to put 2 files.
Title: Re: Code reworking: directory structure
Post by: Antechinus on December 21, 2012, 08:03:40 pm
You gently caress are crazy. :P
Go for 2 levels max.
Title: Re: Code reworking: directory structure
Post by: TestMonkey on December 21, 2012, 08:56:03 pm
Adding a few details...
Tweak on dirs_test4 : https://github.com/norv/Dialogue/tree/dirs_test4/Sources # bring /languages in Sources... (experimental)
So on option #4, we have /Sources with
admin (optional)
controllers
database
ext
languages
lib
Tweak on dirs_test5 : https://github.com/norv/Dialogue/tree/dirs_test5/Sources
# rename, are capitalized names better? (/database and /ext are not modules, neither is exactly the forum core)
In both cases, (big) addons would be probably be like: https://github.com/norv/Dialogue/tree/dirs_test5/Packages/Gallery (pretty much the same structure as a module, no matter if 'taken out' [5] or 'aligned with the rest' [4].)
Title: Re: Code reworking: directory structure
Post by: Spuds on December 22, 2012, 05:59:51 pm
Still think 5 is over the top :D
4 looks good to me ... don't mind the admin functions in their own little area, given that the admin area is its own little thing anyway but not sure about another languages directory under there as well.
I like the rename of the files form Subs-Xyz.php to Xyz.subs.php quite a lot, makes finding them a lot easier to me.
As for the caps on the directories or not ... ummm not sure really ... Maybe caps on our base level and lower for the subs? Today its a mix of, well everything :P
Title: Re: Code reworking: directory structure
Post by: Antechinus on December 22, 2012, 06:50:43 pm
Argggggggggggggggggggghhhhhhhhhhhhh. Just put all languages files in the default/languages directory. Jesus, are you trying to make the whole thing incomprehensible? :P
Title: Re: Code reworking: directory structure
Post by: Spuds on December 22, 2012, 06:57:18 pm
LOL .. I tend to agree ... but it is a fascinating look in to how disturbed Test Monkeys mind is :P
Title: Re: Code reworking: directory structure
Post by: emanuele on December 22, 2012, 06:57:54 pm
To me it's the same. It seemed just a bit odd that if I'm working on Admin-something I have look into a directory for the controller and another one for the "subs".
BTW I think the renaming of Subs-something to something-subs. I was thinking to propose it myself because that way the related files are closer and the alphabetic sorting allows to find things in much less time. :D
Title: Re: Code reworking: directory structure
Post by: TestMonkey on December 23, 2012, 12:05:19 am
The main difference between the two is separation per responsibility first (#4) and separation per feature/functionality first (#5). Well in reality they both exist, the difference is how.
Quick summary: Option #4
it groups per responsibility (controller/subs/templates/languages, with tweaks but mainly this way), and that's the definite way forward, for the next two major versions. There's a lot of work to do for it to be even true.
the area/feature/'module' are distinguished by naming pattern. i.e. Karma., Profile. (and further Profile.History, Profile.Info, etc.), within the core
it doesn't separate first per 'module'/area, but first per responsibility, per role, and only second, per area/functionality.
[big] addons or modules have an example to take (and they will). (they're modules, and within, they separate per responsibility).
there is a correspondence kept (in a measure) between the components... .controller <-> .template should be very good mapping, mostly with <-> .language too... with some exceptions. For .subs it's a bit different, there is some, but they are driven by a different need.
there is and there will be an impedance mismatch between the user interface [to which controller/template map], and the entities, the objects we work with [boards, topics, etc, which is what .subs mostly "follow"]. (i.e. manage boards, boardindex, notifications or read/unread for boards, messageindex, they all deal with "boards" (and may use Boards.subs for their work.)
the library of .subs is not and can never be fully per feature/functionality/user interface-available functionality, but it serves them all (/lib, as it is in option #4). It's loosely the 'M' for 'model', and model doesn't obey to them interface rulez.
core features (I didn't mention this (sorry)), are also candidates for addons/modules. I'll come back on this. We separate them and use (of course) naming patterns and the associated redesign as necessary for them, aka Calendar, Karma [.controller, .template, etc]
I admit... I see other considerents too... The separation per responsibility (if we make it the best) will allow us to look into distributions for Debian or other Linux systems easily... This is also a target for me (still unlikely for first version, we'll see, but as soon as possible).
Title: Re: Code reworking: directory structure
Post by: TestMonkey on December 23, 2012, 04:48:57 am
Quote(10:49:24 AM) Norv: When you make a custom theme and need text strings, do you add a new language file for them, or do you modify one of the core language files? (11:36:54 AM) Antechinus: Depends (11:37:23 AM) Antechinus: If wanting to get it approved at SMF would use a ThemeStrings file (11:37:49 AM) Antechinus: If just for use on a mono-lingual site, would hard code the text (11:38:07 AM) Antechinus: which is easier and is better for performance
Apart from the funny side of it :), none of the options is "modify the files included in Themes/default/languages". Inquiring minds want to know then:
QuoteJust put all languages files in the default/languages directory. Jesus, are you trying to make the whole thing incomprehensible?
What exactly is incomprehensible, about /languages in /Sources? Sources/languages vs Themes/default/languages is a 2 level path vs a 3 level path. ;)
Or, is it incomprehensible if language files are split in admin module? (Sources/admin/languages)
To note: # $languagedir has been defined for a long time. It allows to not hardcode the path to languages (for mods). # to load a language file you use good ole' loadLanguage(). Not write down the path. # there is no issue with adding a language file in a custom theme, no matter if the default main distribution has languages in the theme or it doesn't.
Title: Re: Code reworking: directory structure
Post by: emanuele on December 23, 2012, 04:58:40 am
Quote from: TestMonkey – Apart from the funny side of it :), none of the options is "modify the files included in Themes/default/languages".
Themes installer doesn't allow to do any change to existing files, so even if you want to, you cannot. ;)
Ant wants everything in theme so he can mess with it if he wants! :P
Title: Re: Code reworking: directory structure
Post by: TestMonkey on December 23, 2012, 05:40:48 am
QuoteAs for the caps on the directories or not ... ummm not sure really ... Maybe caps on our base level and lower for the subs? Today its a mix of, well everything :P
To be honest, the only reason why I'd keep the capitalization on base directories is historical.
Title: Re: Code reworking: directory structure
Post by: Antechinus on December 23, 2012, 01:57:05 pm
It looks nicer. :P No, I'm not joking. It really does.
Quote from: TestMonkey – What exactly is incomprehensible, about /languages in /Sources? Sources/languages vs Themes/default/languages is a 2 level path vs a 3 level path. ;)
Or, is it incomprehensible if language files are split in admin module? (Sources/admin/languages)
To note: # $languagedir has been defined for a long time. It allows to not hardcode the path to languages (for mods). # to load a language file you use good ole' loadLanguage(). Not write down the path. # there is no issue with adding a language file in a custom theme, no matter if the default main distribution has languages in the theme or it doesn't.
IF you put languages in Sources then you'll have odd ones out (the theme languages files) or you'll have to make the theme installer write to Sources, which would involve creating extra folders in there for each theme that had a custom file.
You are aware that themes commonly have languages files included even if they don't use ThemeStrings.***.php as such? It's very common to include Settings.****.php in themes. It's rare for them to not have it, so what so you intend to do about that one?
Do you propose to load most but not all languages from Sources, and some but by no means most from individual theme folders? What about the admin centre languages editor? Are you going to make that write to Sources for default themes but write to indivdual theme folders for things like Settings.***.php? More to the point, what is the practical benefit of moving everything if you don't have to? In case you haven't noticed, it already works. :P
Title: Re: Code reworking: directory structure
Post by: Antechinus on December 23, 2012, 01:58:38 pm
The whole idea of having a theme system is so you can mess with stuff. :P
Title: Re: Code reworking: directory structure
Post by: TestMonkey on December 23, 2012, 04:16:44 pm
Ant, I have already answered to some of the questions. And I will say this from the start: if what you are saying is that you think it's a "problem" to set a path to language files and load them from there, you have to be kidding.
First, I will answer per each though. To be clear on them.
Quote from: Antechinus – IF you put languages in Sources then you'll have odd ones out (the theme languages files)
#1. Mods already have what you call "odd ones out": the mods language files. Are they "odd"?
It sounds like habit of thinking to me. Just because you're used to one, you call "odd" a different idea. In case it isn't clear, and it seems by your answers, that it isn't, I'll repeat what I said several times only in this topic: habits of thinking matter too. But don't claim they're grounded in a different actual reasoning, when they are not.
#2 .
Quote from: TestMonkey – To note: # $languagedir has been defined for a long time. It allows to not hardcode the path to languages (for mods).
I will note, customizability of $languagedir is not implemented in the core software, it's there only for mods installer.
Quoteor you'll have to make the theme installer write to Sources, which would involve creating extra folders in there for each theme that had a custom file.
Out of the question for me. Theme installer shouldn't write to /Sources.
QuoteYou are aware that themes commonly have languages files included even if they don't use ThemeStrings.***.php as such? It's very common to include Settings.****.php in themes. It's rare for them to not have it, so what so you intend to do about that one?
Quote from: TestMonkey – # there is no issue with adding a language file in a custom theme, no matter if the default main distribution has languages in the theme or it doesn't.
There is no issue with adding them means there is no issue with the core loading them. We load custom themes from random locations. And by that, we load already /custom_location/languages.
QuoteDo you propose to load most but not all languages from Sources, and some but by no means most from individual theme folders?
Here's what happens today: the core loads most but not all languages from a single location, and some but by no means most from individual theme folders. Here's what happens today: the core loads default language files from a default location, and mods language files from wherever they are, in their own directories. All over the place.
QuoteWhat about the admin centre languages editor? Are you going to make that write to Sources for default themes but write to individual theme folders for things like Settings.***.php?
If your question is about any implementation details, such as pick the right path, then I've already answered several times above.
To summarize: You have to be joking if you're making a fuss over the code choosing a path. We're doing it all the time, you know. We're choosing the path to default_theme/languages from a variable in the database. It doesn't happen by telepathy. :P
IMO, there is a real problem on the last bit you mentioned, but it's different: it is that people might secure their /Sources location more than /Themes. That, yes, it's a concern, and one I wouldn't take lightly. It's implied already in the OP, in the constraint I mentioned. But it's also the other way around: on the contrary, I've been considering to change the directory structure such that we can separate fully files that have to be accessible from the web directory, and those which can be in any location. That's yet another aspect, and the way themes are structured actually makes it more difficult. My next questions here will be: is the feature of editing languages commonly used? Afaik the answer is: no, but it sometimes is. And: do we have a way, to be able to use the 'make files writable only for this session'? The answer is: yes.
The whole idea of having a theme system is so you can mess with stuff. :P
Again, you don't mess with default language files in a custom theme anyway. So this "argument" says something moot.
Sounds like you have the impression that their location would "prevent" you from doing I-don't-know-what-but-it-has-to-be-something-so-Ill-yell, when it does not. (possibly - or not - exception made for the note above).
On the contrary. One of the driving points why I opened this thread is to open for discussion what we can do on the directory structure, for many reasons, including customizability, in particular for addons. Because they are the ones that, by editing directly code and in particular theme code, mess up installations with custom themes and are unable to install and all that yada-yada. Note that the theme system is not the "innocent" here either: the theme installer directly copies files, and the template loading directly overrides the default.
The languages location is a very small point in the big picture of that, a point which I opened for discussion. Reasons for it are related to: separation of the codebase into logical and customizable folders (/languages should be fully customizable, location interchangeable), the example we provide to third-party developers (see the gallery addon example), consistency of directory structure, precisely the attempt to allow more files and folders to be inaccessible from the web directly (albeit in this case a trivial point), and what's more, because internationalization doesn't and shouldn't be dependent on themes (not even on location). (images excepted, obviously). I also have in mind linux distributions, and a couple of others.
Title: Re: Code reworking: directory structure
Post by: Antechinus on December 23, 2012, 05:08:03 pm
Quote from: TestMonkey – #1. Mods already have what you call "odd ones out": the mods language files. Are they "odd"?
Here's what happens today: the core loads most but not all languages from a single location, and some but by no means most from individual theme folders. Here's what happens today: the core loads default language files from a default location, and mods language files from wherever they are, in their own directories. All over the place.
All the mod languages files should be in the Themes/default/languages folder at the moment, or in a sub-folder of that. They shouldn't just be all over the place. If they are, someone has screwed up.
QuoteOut of the question for me. Theme installer shouldn't write to /Sources.
Well you would still need to have the admin centre laguages editor writing to Sources then, and you would presumably still have mods writing to Sources as well. So, if you are thinking of integrating the themes installer and the mod installer, you will still need some way of differentiating theme from mod. Not that it can't be done, just noting it.
QuoteIMO, there is a real problem on the last bit you mentioned, but it's different: it is that people might secure their /Sources location more than /Themes. That, yes, it's a concern, and one I wouldn't take lightly. It's implied already in the OP, in the constraint I mentioned. But it's also the other way around: on the contrary, I've been considering to change the directory structure such that we can separate fully files that have to be accessible from the web directory, and those which can be in any location. That's yet another aspect, and the way themes are structured actually makes it more difficult. My next questions here will be: is the feature of editing languages commonly used? Afaik the answer is: no, but it sometimes is. And: do we have a way, to be able to use the 'make files writable only for this session'? The answer is: yes.
I agree that moving as much out of the web directory as possible isn't a bad idea. The language string editor is quite handy though, so it would be preferable to keep it.
QuoteOn the contrary. One of the driving points why I opened this thread is to open for discussion what we can do on the directory structure, for many reasons, including customizability, in particular for addons. Because they are the ones that, by editing directly code and in particular theme code, mess up installations with custom themes and are unable to install and all that yada-yada. Note that the theme system is not the "innocent" here either: the theme installer directly copies files, and the template loading directly overrides the default.
Yeah sure, but to my mind there is just some stuff that some people will want to do that will not be instantly compatible with other stuff that other people want to do, or may already have done. Past a certain point, some people will need to learn a bit of coding. Things happen.
So, while certainly providing hooks etc to minimise the problems is a good plan, I don't think there's any virtue in getting so paranoid and protective that people are restricted when they do want to do something unusual. If a user doesn't like the result, they don't have to install that theme or mod.
Quote...because internationalization doesn't and shouldn't be dependent on themes (not even on location). (images excepted, obviously). I also have in mind linux distributions, and a couple of others.
Well the i18n will still be dependent, to a minor degree (or not, depending on the themer) upon the theme having the right versions of any custom strings. Don't think there's much that can be done about that.
Side note: that silly old code for the "use image buttons" option. Has that been removed everywhere already? I know it's all over the place in 1.1.x but am not sure if it all went in 2.0.x. If it didn't, it should. Nobody in their right mind will ever use it (CSS is a much better option).
Title: Re: Code reworking: directory structure
Post by: TestMonkey on December 26, 2012, 02:23:59 am
For the record, this is a rough experiment for the following purposes:
make /www directory for publicly accessible files
separate assets from code (recommended practice anyway, and favorite practice by automated builders)
https://github.com/norv/Dialogue/tree/dirs_test7
ETA: the existence of /www/assets directory means it can hold the combined css/js files. (www/assets/hive or www/temp)
Title: Re: Code reworking: directory structure
Post by: TestMonkey on December 26, 2012, 04:04:03 am
Call on capitalization please. As it stands, /Sources, /Themes, and /Packages at a minimum (/Smileys currently too), are capitalized.
Yes or no? (yes = keep capitalization, no = drop, all lowercase)
Last changes: (version #4 with tweaks) https://github.com/norv/Dialogue/tree/dirs_test6
Title: Re: Code reworking: directory structure
Post by: TestMonkey on December 26, 2012, 05:17:44 am
I'd like to wrap up this discussion. There isn't a full consensus. But, lets see what we can do. I will state from the start though: in the #4 vs #5 choice, I think #4 with some tweaks (known as #6) is more appropriate.
One point, in support of option #4, heading towards a different way to look at option #5:
Still in mind with the next two major versions: one important thing we are pursuing, from 1.0, is introduction/reworking of mechanisms to faciliate plugging in seamlessly (a big "subsite" addon, as well as a database layer, as well as a core feature). Gedanken experiment: add a /addons directory to the root, and have Gallery, [Social]Profile and Calendar in it. They all build your site. => a somewhat similar result with both "options". We already have mechanisms to load seamlessly (atm limited to loading from /sourcedir but that will change). https://github.com/norv/Dialogue/tree/dirs_test6
It's built on option #4 with a few tweaks. All addons we are working on (official) have their place for all their files.
IMO option #4 (#6) is clean and appropriate. For the codebase, what we work on it, where we move it - please keep in mind we visually separate 'Component'-'Area too.
More importantly... how do I put this. I will state my personal perspective. Directory structure is not alpha and omega, but it matters, and for the moment (given the codebase status and way we work with it, too) I do give it a particular significance:
Title: Re: Code reworking: directory structure
Post by: Spuds on December 26, 2012, 11:06:24 am
Number 6, the test formally known as 4, looks good to me
I thought number 5 was my least favorite until I saw number 7 .... I dislike 5, I abhor 7
I'd guess I'd vote for all lowercase at this point as I can't come up with a good reason (other than legacy) for leaving just a handful of them like that with the rest all lower.
Title: Re: Code reworking: directory structure
Post by: Antechinus on December 26, 2012, 03:18:01 pm
Gotta admit I prefer capitalised just for aesthetics, but it's not a big deal to me either way.
Title: Re: Code reworking: directory structure
Post by: IchBin on December 28, 2012, 12:47:13 am
Agree with the Potato man here. I like option 6 the best. Number 7 and 5 need to die by fire.